Data Factory

Official Documentation

Service Description

Data processing in today's companies is marked by heterogeneous data storage (SQL, NoSQL, unstructured data, etc.) and processing components (databases, Big Data processors, etc.). Data in a company often passes through complex paths from generation or receipt of the data, through various data processing components, to storage or distribution of the data to various recipients. With Data Factory, local data such as that from SQL Server can be processed together with cloud-related data from Azure SQL Database, Blobs, and Tables. These data processing streams can be created, processed, and monitored through simple, highly available data pipelines. Data sources and data recipients can be defined, and the movement of the data in the company can be traced and monitored from a central location.

Getting Started

  1. 10/7/2015, Video, 0:59:31
    The data landscape is more varied than ever with unstructured and structured data originating from many cloud and on-premises sources. Data Factory enables you to process...
  2. 2/25/2016, Mva
    Exploring data orchestration concepts? Check out this course on the basic capabilities of Azure Data Factory (ADF). Get an overview of advanced analytics, and see how Azure...
  3. 1/16/2017, Mva
    If you’d like to learn how to architect solutions in Cortana Intelligence Suite and how to build intelligence into your applications, don’t miss this workshop! Build an...
  4. 10/7/2015, Video, 0:31:34
    This video takes a deeper dive into the features and functions of the Azure Data Factory orchestration engineLearn more: http://aka.ms/g7hlpt
  5. 10/7/2015, Video, 1:11:26
    This video takes a deeper dive into how Azure Data Factory can be used to build hybrid big data analytics pipelines through the lens of an automated risk processing use case...



Latest Content

Subscribe to News about Data Factory

Title  
Blog
Blog
Video
Video
Video
Video
Video
Video
Video
Video
Blog
Video
more...


Web Content

Data Factory Documentation

1. Switch to version 1 documentation
2. Overview
     2.1. Introduction to Data Factory
     2.2. Compare current version to version 1
3. Quickstarts
     3.1. Create data factory - User interface (UI)
     3.2. Create data factory - Copy Data tool
     3.3. Create data factory - Azure PowerShell
     3.4. Create data factory - .NET
     3.5. Create data factory - Python
     3.6. Create data factory - REST
     3.7. Create data factory - Resource Manager template
4. Tutorials
     4.1. Copy data in cloud
          4.1.1. Copy Data tool
          4.1.2. User interface (UI)
          4.1.3. .NET
     4.2. Copy on-premises data to cloud
          4.2.1. Copy Data tool
          4.2.2. User interface (UI)
          4.2.3. Azure PowerShell
     4.3. Copy data in bulk
          4.3.1. User interface (UI)
          4.3.2. Azure PowerShell
     4.4. Copy data incrementally
          4.4.1. 1 - Copy from one table
               4.4.1.1. User interface (UI)
               4.4.1.2. Azure PowerShell
          4.4.2. 2 - Copy from multiple tables
               4.4.2.1. User interface (UI)
               4.4.2.2. Azure PowerShell
          4.4.3. 3 - Use change tracking feature
               4.4.3.1. User interface (UI)
               4.4.3.2. Azure PowerShell
     4.5. Transform data in cloud
          4.5.1. HDInsight Spark
               4.5.1.1. User interface (UI)
               4.5.1.2. Azure PowerShell
          4.5.2. Databricks Notebook
               4.5.2.1. User interface (UI)
     4.6. Transform data in virtual network
          4.6.1. User interface (UI)
          4.6.2. Azure PowerShell
     4.7. Add branching and chaining
          4.7.1. User interface (UI)
          4.7.2. .NET
     4.8. Run SSIS packages in Azure
          4.8.1. User interface (UI)
          4.8.2. Azure PowerShell
5. Samples
     5.1. Code samples
     5.2. Azure PowerShell
6. Concepts
     6.1. Pipelines and activities
     6.2. Datasets and linked services
     6.3. Pipeline execution and triggers
     6.4. Integration runtime
     6.5. Roles and permissions
     6.6. Understanding pricing
     6.7. Naming rules
7. How-to guides
     7.1. Author
          7.1.1. Visually author data factories
          7.1.2. Continuous integration and delivery
          7.1.3. Iterative development and debugging
     7.2. Connectors
          7.2.1. Amazon Marketplace Web Service
          7.2.2. Amazon Redshift
          7.2.3. Amazon S3
          7.2.4. Azure Blob Storage
          7.2.5. Azure Cosmos DB
          7.2.6. Azure Data Lake Storage Gen1
          7.2.7. Azure Data Lake Storage Gen2
          7.2.8. Azure Database for MySQL
          7.2.9. Azure Database for PostgreSQL
          7.2.10. Azure File Storage
          7.2.11. Azure Search
          7.2.12. Azure SQL Database
          7.2.13. Azure SQL Database Managed Instance
          7.2.14. Azure SQL Data Warehouse
          7.2.15. Azure Table Storage
          7.2.16. Cassandra
          7.2.17. Common Data Service for Apps
          7.2.18. Concur
          7.2.19. Couchbase
          7.2.20. DB2
          7.2.21. Drill
          7.2.22. Dynamics 365
          7.2.23. Dynamics AX
          7.2.24. Dynamics CRM
          7.2.25. File System
          7.2.26. FTP
          7.2.27. Google AdWords
          7.2.28. Google BigQuery
          7.2.29. Greenplum
          7.2.30. HBase
          7.2.31. HDFS
          7.2.32. Hive
          7.2.33. HTTP
          7.2.34. HubSpot
          7.2.35. Impala
          7.2.36. Informix
          7.2.37. Jira
          7.2.38. Magento
          7.2.39. MariaDB
          7.2.40. Marketo
          7.2.41. Microsoft Access
          7.2.42. MongoDB
          7.2.43. MySQL
          7.2.44. Netezza
          7.2.45. OData
          7.2.46. ODBC
          7.2.47. Office 365
          7.2.48. Oracle
          7.2.49. Oracle Eloqua
          7.2.50. Oracle Responsys
          7.2.51. Oracle Service Cloud
          7.2.52. Paypal
          7.2.53. Phoenix
          7.2.54. PostgreSQL
          7.2.55. Presto
          7.2.56. QuickBooks Online
          7.2.57. Salesforce
          7.2.58. Salesforce Service Cloud
          7.2.59. Salesforce Marketing Cloud
          7.2.60. SAP Business Warehouse
          7.2.61. SAP Cloud for Customer
          7.2.62. SAP ECC
          7.2.63. SAP HANA
          7.2.64. ServiceNow
          7.2.65. SFTP
          7.2.66. Shopify
          7.2.67. Spark
          7.2.68. SQL Server
          7.2.69. Square
          7.2.70. Sybase
          7.2.71. Teradata
          7.2.72. Vertica
          7.2.73. Web Table
          7.2.74. Xero
          7.2.75. Zoho
     7.3. Copy data
          7.3.1. Copy data using Copy Activity
          7.3.2. Copy Data tool
          7.3.3. Load Data Lake Storage Gen2
               7.3.3.1. Copy from Data Lake Storage Gen1
          7.3.4. Load SQL Data Warehouse
          7.3.5. Load Data Lake Storage Gen1
          7.3.6. Load Office 365 data
          7.3.7. Read or write partitioned data
          7.3.8. Format and compression support
          7.3.9. Schema and type mapping
          7.3.10. Fault tolerance
          7.3.11. Performance and tuning
     7.4. Transform data
          7.4.1. HdInsight Hive Activity
          7.4.2. HdInsight Pig Activity
          7.4.3. HdInsight MapReduce Activity
          7.4.4. HdInsight Streaming Activity
          7.4.5. HdInsight Spark Activity
          7.4.6. ML Batch Execution Activity
          7.4.7. ML Update Resource Activity
          7.4.8. Stored Procedure Activity
          7.4.9. Data Lake U-SQL Activity
          7.4.10. Databricks Notebook Activity
          7.4.11. Databricks Jar Activity
          7.4.12. Databricks Python Activity
          7.4.13. Custom activity
          7.4.14. Compute linked services
     7.5. Control flow
          7.5.1. Append Variable Activity
          7.5.2. Azure Function Activity
          7.5.3. Execute Pipeline Activity
          7.5.4. Filter Activity
          7.5.5. For Each Activity
          7.5.6. Get Metadata Activity
          7.5.7. If Condition Activity
          7.5.8. Lookup Activity
          7.5.9. Set Variable Activity
          7.5.10. Until Activity
          7.5.11. Wait Activity
          7.5.12. Web Activity
     7.6. Parameterize
          7.6.1. Parameterize linked services
          7.6.2. Expression Language
          7.6.3. System variables
     7.7. Security
          7.7.1. Data movement security considerations
          7.7.2. Store credentials in Azure Key Vault
          7.7.3. Encrypt credentials for self-hosted integration runtime
          7.7.4. Data factory service identity
     7.8. Monitor and manage
          7.8.1. Monitor visually
          7.8.2. Monitor with Azure Monitor
          7.8.3. Monitor with SDKs
          7.8.4. Monitor integration runtime
          7.8.5. Monitor Azure-SSIS integration runtime
          7.8.6. Reconfigure Azure-SSIS integration runtime
     7.9. Create integration runtime
          7.9.1. Azure integration runtime
          7.9.2. Self hosted integration runtime
          7.9.3. Azure-SSIS integration runtime
          7.9.4. Shared self-hosted integration runtime
     7.10. Run SSIS packages in Azure
          7.10.1. Run SSIS packages with Execute SSIS Package activity
          7.10.2. Run SSIS packages with Stored Procedure activity
          7.10.3. Schedule Azure-SSIS integration runtime
          7.10.4. Join Azure-SSIS IR to a virtual network
          7.10.5. Enable Azure AD authentication for Azure-SSIS IR
          7.10.6. Provision Enterprise Edition for Azure-SSIS IR
          7.10.7. Customize setup for Azure-SSIS IR
          7.10.8. Install licensed components for Azure-SSIS IR
          7.10.9. Configure high performance for Azure-SSIS IR
          7.10.10. Configure disaster recovery for Azure-SSIS IR
          7.10.11. Clean up SSISDB logs with Elastic Database Jobs
     7.11. Create triggers
          7.11.1. Create a schedule trigger
          7.11.2. Create a tumbling window trigger
          7.11.3. Create an event-based trigger
8. Reference
     8.1. .NET
     8.2. PowerShell
     8.3. REST API
     8.4. Python
9. Resources
     9.1. Ask a question - MSDN forum
     9.2. Ask a question - Stack Overflow
     9.3. Request a feature
     9.4. FAQ
     9.5. Roadmap
     9.6. Pricing
     9.7. Availability by region
     9.8. Support options

Online Training Content

Date Title
5/24/2017 Orchestrating Big Data with Azure Data Factory
1/16/2017 Cortana Intelligence Suite End-to-End
7/4/2016 Design and Implement Big Data & Advanced Analytics Solutions
2/25/2016 Orchestrating Data and Services with Azure Data Factory

Tools

Tool Description

Videos

Date Title Length
8/3/2018
Execute Jars and Python scripts on Azure Databricks using Data Factory | Azure Friday
0:10:51
8/1/2018
Azure Data Factory new features and integration with Azure Databricks | Data Exposed
0:16:19
8/1/2018
Azure Data Factory new features and integration with Azure Databricks
0:13:41
7/21/2018
Event-based data integration with Azure Data Factory | Azure Friday
0:09:57
7/16/2018
How to develop and debug with Azure Data Factory | Azure Friday
0:07:40
5/12/2018
Iterative development and debugging with Azure Data Factory
0:07:38
5/11/2018
New capabilities for modern data integration in the cloud : Build 2018
1:19:23
5/10/2018
ETL 2.0 - Data Engineering for developers  : Build 2018
1:29:13
5/9/2018
Develop scalable analytical solutions with Azure Data Factory & Azure SQL Data Warehouse
1:00:15
5/7/2018
Ingest, prepare & transform using Azure Databricks & Data Factory | Azure Friday
0:11:05

Page 3 of 8