Data Factory

Official Documentation

Service Description

Data processing in today's companies is marked by heterogeneous data storage (SQL, NoSQL, unstructured data, etc.) and processing components (databases, Big Data processors, etc.). Data in a company often passes through complex paths from generation or receipt of the data, through various data processing components, to storage or distribution of the data to various recipients. With Data Factory, local data such as that from SQL Server can be processed together with cloud-related data from Azure SQL Database, Blobs, and Tables. These data processing streams can be created, processed, and monitored through simple, highly available data pipelines. Data sources and data recipients can be defined, and the movement of the data in the company can be traced and monitored from a central location.

Getting Started

  1. 10/7/2015, Video, 0:59:31
    The data landscape is more varied than ever with unstructured and structured data originating from many cloud and on-premises sources. Data Factory enables you to process...
  2. 2/25/2016, Mva
    Exploring data orchestration concepts? Check out this course on the basic capabilities of Azure Data Factory (ADF). Get an overview of advanced analytics, and see how Azure...
  3. 1/16/2017, Mva
    If you’d like to learn how to architect solutions in Cortana Intelligence Suite and how to build intelligence into your applications, don’t miss this workshop! Build an...
  4. 10/7/2015, Video, 0:31:34
    This video takes a deeper dive into the features and functions of the Azure Data Factory orchestration engineLearn more: http://aka.ms/g7hlpt
  5. 10/7/2015, Video, 1:11:26
    This video takes a deeper dive into how Azure Data Factory can be used to build hybrid big data analytics pipelines through the lens of an automated risk processing use case...



Latest Content

Subscribe to News about Data Factory

Title  
Video
Blog
Blog
Video
Video
Blog
Video
Blog
Blog
Video
Video
Video
more...


Azure Documentation

1. Data Factory V2 Documentation
2. Overview
     2.1. Introduction to Data Factory
     2.2. Compare versions 1 and 2
3. Quickstarts
     3.1. Create data factory
          3.1.1. PowerShell
          3.1.2. .NET
          3.1.3. Python
          3.1.4. REST
          3.1.5. Portal
          3.1.6. Resource Manager template
4. Tutorials
     4.1. 1 - Deploy SSIS packages to Azure
     4.2. 2 - Copy data in cloud
     4.3. 3 - Copy on-premises data to cloud
     4.4. 4 - Copy data in bulk
     4.5. 5 - Copy data incrementally
          4.5.1. 5.1 - Copy from one table
          4.5.2. 5.2 - Copy from multiple tables
          4.5.3. 5.3 - Use change tracking feature
     4.6. 6 - Transform data in cloud
     4.7. 7 - Transform data in virtual network
     4.8. 8 - Control flow
5. Samples
     5.1. Code samples
     5.2. PowerShell
6. Concepts
     6.1. Pipelines and activities
     6.2. Datasets and linked services
     6.3. Pipeline execution and triggers
     6.4. Integration runtime
7. How-to guides
     7.1. Copy data
          7.1.1. Copy data from supported sources to destinations
          7.1.2. Read or write partitioned data
          7.1.3. Connectors
               7.1.3.1. Amazon Marketplace Web Service
               7.1.3.2. Amazon Redshift
               7.1.3.3. Amazon S3
               7.1.3.4. Azure Blob Storage
               7.1.3.5. Azure Cosmos DB
               7.1.3.6. Azure Data Lake Store
               7.1.3.7. Azure Database for MySQL
               7.1.3.8. Azure Database for PostgreSQL
               7.1.3.9. Azure File Storage
               7.1.3.10. Azure Search
               7.1.3.11. Azure SQL Database
               7.1.3.12. Azure SQL Data Warehouse
               7.1.3.13. Azure Table Storage
               7.1.3.14. Cassandra
               7.1.3.15. Concur
               7.1.3.16. Couchbase
               7.1.3.17. DB2
               7.1.3.18. Drill
               7.1.3.19. Dynamics 365
               7.1.3.20. Dynamics CRM
               7.1.3.21. File System
               7.1.3.22. FTP
               7.1.3.23. GE Historian
               7.1.3.24. Google BigQuery
               7.1.3.25. Greenplum
               7.1.3.26. HBase
               7.1.3.27. HDFS
               7.1.3.28. Hive
               7.1.3.29. HTTP
               7.1.3.30. HubSpot
               7.1.3.31. Impala
               7.1.3.32. Informix
               7.1.3.33. Jira
               7.1.3.34. Magento
               7.1.3.35. MariaDB
               7.1.3.36. Marketo
               7.1.3.37. Microsoft Access
               7.1.3.38. MongoDB
               7.1.3.39. MySQL
               7.1.3.40. OData
               7.1.3.41. ODBC
               7.1.3.42. Oracle
               7.1.3.43. Oracle Eloqua
               7.1.3.44. Paypal
               7.1.3.45. Phoenix
               7.1.3.46. PostgreSQL
               7.1.3.47. Presto
               7.1.3.48. QuickBooks
               7.1.3.49. Salesforce
               7.1.3.50. Salesforce Service Cloud
               7.1.3.51. SAP Business Warehouse
               7.1.3.52. SAP Cloud for Customer
               7.1.3.53. SAP HANA
               7.1.3.54. ServiceNow
               7.1.3.55. SFTP
               7.1.3.56. Shopify
               7.1.3.57. Spark
               7.1.3.58. SQL Server
               7.1.3.59. Square
               7.1.3.60. Sybase
               7.1.3.61. Teradata
               7.1.3.62. Web Table
               7.1.3.63. Xero
               7.1.3.64. Zoho
          7.1.4. Format and compression support
          7.1.5. Schema and type mapping
          7.1.6. Fault tolerance
          7.1.7. Performance and tuning
     7.2. Transform data
          7.2.1. HdInsight Hive Activity
          7.2.2. HdInsight Pig Activity
          7.2.3. HdInsight MapReduce Activity
          7.2.4. HdInsight Streaming Activity
          7.2.5. HdInsight Spark Activity
          7.2.6. ML Batch Execution Activity
          7.2.7. ML Update Resource Activity
          7.2.8. Stored Procedure Activity
          7.2.9. Data Lake U-SQL Activity
          7.2.10. Custom activity
          7.2.11. Compute linked services
     7.3. Control flow
          7.3.1. If Condition Activity
          7.3.2. Until Activity
          7.3.3. Web Activity
          7.3.4. Lookup Activity
          7.3.5. Get Metadata Activity
          7.3.6. Execute Pipeline Activity
          7.3.7. For Each Activity
          7.3.8. Wait Activity
          7.3.9. Expression Language
          7.3.10. System variables
     7.4. Security
          7.4.1. Data movement security considerations
          7.4.2. Store credentials in Azure Key Vault
          7.4.3. Encrypt credentials for self-hosted integration runtime
          7.4.4. Data factory service identity
     7.5. Monitor and manage
          7.5.1. Use SDKs
          7.5.2. Use Azure Monitor
          7.5.3. Monitor visually
          7.5.4. Manage Azure-SSIS integration runtime
          7.5.5. Monitor integration runtime
     7.6. Create integration runtime
          7.6.1. Azure integration runtime
          7.6.2. Self hosted integration runtime
          7.6.3. Azure-SSIS integration runtime
     7.7. Run SSIS packages in Azure
          7.7.1. Create Azure-SSIS integration runtime
          7.7.2. Join Azure-SSIS integration runtime to a VNET
          7.7.3. Manage Azure-SSIS integration runtime
          7.7.4. Configure performance settings of Azure-SSIS integration runtime
          7.7.5. Invoke SSIS packages using stored procedure activity
     7.8. Create triggers
          7.8.1. Create a schedule trigger
          7.8.2. Create a tumbling window trigger
8. Reference
     8.1. .NET
     8.2. PowerShell
     8.3. REST API
     8.4. Python
9. Resources
     9.1. FAQ
     9.2. Azure Roadmap
     9.3. Pricing
     9.4. MSDN forum
     9.5. Stack Overflow
     9.6. Request a feature
     9.7. Region availability
     9.8. Support options

Online Training Content

Date Title
5/24/2017 Orchestrating Big Data with Azure Data Factory
1/16/2017 Cortana Intelligence Suite End-to-End
7/4/2016 Design and Implement Big Data & Advanced Analytics Solutions
2/25/2016 Orchestrating Data and Services with Azure Data Factory

Tools

Tool Description

Videos

Date Title Length
1/16/2018 Azure Friday | Visually build pipelines for Azure Data Factory V2 0:19:32
11/17/2017 Lift and shift SSIS to the cloud by using Azure Data Factory | T138 0:09:01
11/15/2017 Lift and shift SSIS to the cloud by using Azure Data Factory 0:09:00
10/23/2017 Cloud Tech 10 - 23rd October 2017 - AWS Cloud Service Map, Bing Custom Search, Cosmos DB and more! 0:04:26
9/30/2017 Deep dive into SQL Server Integration Services (SSIS) 2017 and beyond 1:25:20
9/30/2017 Getting peak performance from your SQL Data Warehouse column store 1:15:23
9/28/2017 New capabilities for data integration in the cloud 1:35:27
4/2/2017 Cloud Tech 10 - 3rd April 2017 0:08:48
3/27/2017 Cloud Tech 10 - 27th March 2017 0:08:37
3/13/2017 Cloud Tech 10 - 13th March 2017 0:09:06

Page 1 of 4