Azure Data Factory
Hybrid data integration at enterprise scale, made easy.
Web Resources
Connectivity from IaaS(VM) and PaaS service(ADF) to ADLS account with public network access disabled
In this blog post, we will investigate the impacts of disabling public network access in the storage account and learn how to establish connections to an Azure Data Lake Storage account from both IaaS (Virtual Machine) and PaaS services (Azure Data Factory), when public network access is disabled. Additionally, we will address common troubleshooting issues that may arise due to this configuration.
Introduction
Azure storage accounts provide a scalable and secure way to store and access data in the cloud. This model enables you to control the level of access to your storage accounts that...
The blog aims to perform simple File storage REST API operations such as Create, Copy, Rename and Delete operations using CURL utility.Let’s look at some of the CURL command syntax to perform REST API operations and will be making use of SAS as the auth scheme.
We need to take care of the pointers below while performing the operations via CURL:
Ensure the URL is formed correctly as per the operation you are trying to perform.
The mandatory header needs to be passed and can be done using “-H” parameter.
Ensure you are appending/removing extra ‘?’ to the SAS token in the URLs...
The blog is inclined towards troubleshooting clientothererrors reflecting in the metrics. It is a step-by-step process to understand what these errors signify and potential reasons. This will also help in analyzing if these are legitimate ones and also deciding on the action plan ahead.
ClientOtherError usually means expected client-side errors which are not fatal errors. These operations have been completed successfully and therefore don't affect other metrics, such as availability. Some examples of operations that execute successfully but that can result in unsuccessful HTTP status...
Symptoms:
Uploading files using Java SDK class DataLakeFileClient fails to initiate the ADF trigger.
Uploading files using Azure Portal initiates the trigger as expected.
ADF trigger is configured to ignore empty blobs.
Analysis:
Whenever a file is created using ADLS Gen2 REST API, there are 3 operations called at the backend i.e. CreateFile, AppendFile and FlushFile.
Microsoft.Storage.BlobCreated event is triggered when a blob is created or replaced. Specifically, this event is triggered when clients use the CreateFile and FlushWithClose operations that are available...
Background:
Many times, we have a requirement to delete the older logs from $logs as the Retention days as it was not supported for ADLS gen2 storage account.
Please note Product team already worked on this fix and in case automatic deletion is not working can follow the below steps
Please refer to article https://docs.microsoft.com/en-us/azure/storage/common/manage-storage-analytics-logs?tabs=azure-portal#modify-log-data-retention-period for more information
Deletion of logs from the $logs can be achieved from the following options
Azure Storage explorer
REST...