Data Factory

Official Documentation

Service Description

Data processing in today's companies is marked by heterogeneous data storage (SQL, NoSQL, unstructured data, etc.) and processing components (databases, Big Data processors, etc.). Data in a company often passes through complex paths from generation or receipt of the data, through various data processing components, to storage or distribution of the data to various recipients. With Data Factory, local data such as that from SQL Server can be processed together with cloud-related data from Azure SQL Database, Blobs, and Tables. These data processing streams can be created, processed, and monitored through simple, highly available data pipelines. Data sources and data recipients can be defined, and the movement of the data in the company can be traced and monitored from a central location.

Getting Started

  1. 10/7/2015, Video, 0:59:31
    The data landscape is more varied than ever with unstructured and structured data originating from many cloud and on-premises sources. Data Factory enables you to process...
  2. 2/25/2016, Mva
    Exploring data orchestration concepts? Check out this course on the basic capabilities of Azure Data Factory (ADF). Get an overview of advanced analytics, and see how Azure...
  3. 1/16/2017, Mva
    If you’d like to learn how to architect solutions in Cortana Intelligence Suite and how to build intelligence into your applications, don’t miss this workshop! Build an...
  4. 10/7/2015, Video, 0:31:34
    This video takes a deeper dive into the features and functions of the Azure Data Factory orchestration engineLearn more: http://aka.ms/g7hlpt
  5. 10/7/2015, Video, 1:11:26
    This video takes a deeper dive into how Azure Data Factory can be used to build hybrid big data analytics pipelines through the lens of an automated risk processing use case...



Latest Content

Subscribe to News about Data Factory

Title  
Blog
Podcast
Blog
Blog
Podcast
Blog
Blog
Blog
Blog
Blog
Video
Blog
more...


Web Content

Azure Documentation

1. Data Factory Documentation
2. Switch to version 1 documentation
3. Overview
     3.1. Introduction to Data Factory
     3.2. Compare current version to version 1
4. Quickstarts
     4.1. Create data factory - User interface (UI)
     4.2. Create data factory - Copy Data tool
     4.3. Create data factory - Azure PowerShell
     4.4. Create data factory - .NET
     4.5. Create data factory - Python
     4.6. Create data factory - REST
     4.7. Create data factory - Resource Manager template
5. Tutorials
     5.1. 1 - Provision Azure-SSIS integration runtime
          5.1.1. User interface (UI)
          5.1.2. Azure PowerShell
     5.2. 2 - Copy data in cloud
          5.2.1. Copy Data tool
          5.2.2. User interface (UI)
          5.2.3. .NET
     5.3. 3 - Copy on-premises data to cloud
          5.3.1. Copy Data tool
          5.3.2. User interface (UI)
          5.3.3. Azure PowerShell
     5.4. 4 - Copy data in bulk
          5.4.1. User interface (UI)
          5.4.2. Azure PowerShell
     5.5. 5 - Copy data incrementally
          5.5.1. 5.1 - Copy from one table
               5.5.1.1. User interface (UI)
               5.5.1.2. Azure PowerShell
          5.5.2. 5.2 - Copy from multiple tables
               5.5.2.1. User interface (UI)
               5.5.2.2. Azure PowerShell
          5.5.3. 5.3 - Use change tracking feature
               5.5.3.1. User interface (UI)
               5.5.3.2. Azure PowerShell
     5.6. 6 - Transform data in cloud
          5.6.1. HDInsight Spark
               5.6.1.1. User interface (UI)
               5.6.1.2. Azure PowerShell
          5.6.2. Databricks Notebook
               5.6.2.1. User interface (UI)
     5.7. 7 - Transform data in virtual network
          5.7.1. User interface (UI)
          5.7.2. Azure PowerShell
     5.8. 8 - Control flow
          5.8.1. User interface (UI)
          5.8.2. .NET
6. Samples
     6.1. Code samples
     6.2. Azure PowerShell
7. Concepts
     7.1. Pipelines and activities
     7.2. Datasets and linked services
     7.3. Pipeline execution and triggers
     7.4. Integration runtime
8. How-to guides
     8.1. Author
          8.1.1. Visually author data factories
          8.1.2. Continuous integration and deployment
          8.1.3. Iterative development and debugging
     8.2. Connectors
          8.2.1. Amazon Marketplace Web Service
          8.2.2. Amazon Redshift
          8.2.3. Amazon S3
          8.2.4. Azure Blob Storage
          8.2.5. Azure Cosmos DB
          8.2.6. Azure Data Lake Storage Gen1
          8.2.7. Azure Data Lake Storage Gen2
          8.2.8. Azure Database for MySQL
          8.2.9. Azure Database for PostgreSQL
          8.2.10. Azure File Storage
          8.2.11. Azure Search
          8.2.12. Azure SQL Database
          8.2.13. Azure SQL Data Warehouse
          8.2.14. Azure Table Storage
          8.2.15. Cassandra
          8.2.16. Common Data Service for Apps
          8.2.17. Concur
          8.2.18. Couchbase
          8.2.19. DB2
          8.2.20. Drill
          8.2.21. Dynamics 365
          8.2.22. Dynamics CRM
          8.2.23. File System
          8.2.24. FTP
          8.2.25. GE Historian
          8.2.26. Google BigQuery
          8.2.27. Greenplum
          8.2.28. HBase
          8.2.29. HDFS
          8.2.30. Hive
          8.2.31. HTTP
          8.2.32. HubSpot
          8.2.33. Impala
          8.2.34. Informix
          8.2.35. Jira
          8.2.36. Magento
          8.2.37. MariaDB
          8.2.38. Marketo
          8.2.39. Microsoft Access
          8.2.40. MongoDB
          8.2.41. MySQL
          8.2.42. Netezza
          8.2.43. OData
          8.2.44. ODBC
          8.2.45. Oracle
          8.2.46. Oracle Eloqua
          8.2.47. Oracle Responsys
          8.2.48. Paypal
          8.2.49. Phoenix
          8.2.50. PostgreSQL
          8.2.51. Presto
          8.2.52. QuickBooks
          8.2.53. Salesforce
          8.2.54. Salesforce Service Cloud
          8.2.55. Salesforce Marketing Cloud
          8.2.56. SAP Business Warehouse
          8.2.57. SAP Cloud for Customer
          8.2.58. SAP ECC
          8.2.59. SAP HANA
          8.2.60. ServiceNow
          8.2.61. SFTP
          8.2.62. Shopify
          8.2.63. Spark
          8.2.64. SQL Server
          8.2.65. Square
          8.2.66. Sybase
          8.2.67. Teradata
          8.2.68. Vertica
          8.2.69. Web Table
          8.2.70. Xero
          8.2.71. Zoho
     8.3. Copy data
          8.3.1. Copy data using Copy Activity
          8.3.2. Copy Data tool
          8.3.3. Load Data Lake Storage Gen2
               8.3.3.1. Copy from Data Lake Storage Gen1
          8.3.4. Load SQL Data Warehouse
          8.3.5. Load Data Lake Storage Gen1
          8.3.6. Read or write partitioned data
          8.3.7. Format and compression support
          8.3.8. Schema and type mapping
          8.3.9. Fault tolerance
          8.3.10. Performance and tuning
     8.4. Transform data
          8.4.1. HdInsight Hive Activity
          8.4.2. HdInsight Pig Activity
          8.4.3. HdInsight MapReduce Activity
          8.4.4. HdInsight Streaming Activity
          8.4.5. HdInsight Spark Activity
          8.4.6. ML Batch Execution Activity
          8.4.7. ML Update Resource Activity
          8.4.8. Stored Procedure Activity
          8.4.9. Data Lake U-SQL Activity
          8.4.10. Databricks Notebook Activity
          8.4.11. Databricks Jar Activity
          8.4.12. Databricks Python Activity
          8.4.13. Custom activity
          8.4.14. Compute linked services
     8.5. Control flow
          8.5.1. Execute Pipeline Activity
          8.5.2. Filter Activity
          8.5.3. For Each Activity
          8.5.4. Get Metadata Activity
          8.5.5. If Condition Activity
          8.5.6. Lookup Activity
          8.5.7. Until Activity
          8.5.8. Wait Activity
          8.5.9. Web Activity
          8.5.10. Expression Language
          8.5.11. System variables
     8.6. Security
          8.6.1. Data movement security considerations
          8.6.2. Store credentials in Azure Key Vault
          8.6.3. Encrypt credentials for self-hosted integration runtime
          8.6.4. Data factory service identity
     8.7. Monitor and manage
          8.7.1. Monitor visually
          8.7.2. Monitor with Azure Monitor
          8.7.3. Monitor with OMS
          8.7.4. Monitor with SDKs
          8.7.5. Monitor integration runtime
          8.7.6. Monitor Azure-SSIS integration runtime
          8.7.7. Reconfigure Azure-SSIS integration runtime
     8.8. Create integration runtime
          8.8.1. Azure integration runtime
          8.8.2. Self hosted integration runtime
          8.8.3. Azure-SSIS integration runtime
     8.9. Run SSIS packages in Azure
          8.9.1. Join Azure-SSIS IR to a virtual network
          8.9.2. Enable Azure AD authentication for Azure-SSIS IR
          8.9.3. Configure high performance for Azure-SSIS IR
          8.9.4. Provision Enterprise Edition for Azure-SSIS IR
          8.9.5. Customize setup for Azure-SSIS IR
          8.9.6. Install licensed components for Azure-SSIS IR
          8.9.7. Run SSIS packages with Execute SSIS Package activity
          8.9.8. Run SSIS packages with Stored Procedure activity
          8.9.9. Schedule Azure-SSIS integration runtime
     8.10. Create triggers
          8.10.1. Create a schedule trigger
          8.10.2. Create a tumbling window trigger
          8.10.3. Create an event-based trigger
9. Reference
     9.1. .NET
     9.2. PowerShell
     9.3. REST API
     9.4. Python
10. Resources
     10.1. Ask a question - MSDN forum
     10.2. Ask a question - Stack Overflow
     10.3. Request a feature
     10.4. FAQ
     10.5. Roadmap
     10.6. Pricing
     10.7. Availability by region
     10.8. Support options

Online Training Content

Date Title
5/24/2017 Orchestrating Big Data with Azure Data Factory
1/16/2017 Cortana Intelligence Suite End-to-End
7/4/2016 Design and Implement Big Data & Advanced Analytics Solutions
2/25/2016 Orchestrating Data and Services with Azure Data Factory

Tools

Tool Description

Videos

Date Title Length
2/10/2017
Orchestrating Big Data Pipelines with Azure Data Factory
1:15:24
2/10/2017
Interview with Lace Lofranco
1:02:06
7/13/2016
Advancements in Data Technology
0:29:42
7/12/2016
Hadoop without borders – learn to build hybrid big data analytics pipelines with Azure Data Factory
0:58:50
6/29/2016
Azure Data Factory - Monitoring and Managing Big Data Pipelines
0:11:54
6/7/2016
Azure Data Factory - Monitoring and Managing Big Data Piplines
0:11:53
5/5/2016
Data Factory Concepts
0:06:13
4/6/2016
How to Monitor & Manage Big Data Pipelines with Azure Data Factory
1:01:40
3/30/2016
Data Integration in the Cloud and Building Data Analytics Pipelines
0:33:03
2/3/2016
Azure Data Lake Analytics Deep Dive
0:59:43

Page 3 of 5