I’m trying to import dags from a blob storage to a managed airflow from Azure Data Factory. The ADF instance can access the blob storage through the System Managed Identity (I can browse the storage through ADF), but when i try to upload the dags, I have the following issue:
Microsoft.WindowsAzure.Storage.StorageException: This request is not authorized to perform this operation.
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.ListBlobsSegmentedAsync(String prefix, Boolean useFlatBlobListing, BlobListingDetails blobListingDetails, Nullable`1 maxResults, BlobContinuationToken currentToken, BlobRequestOptions options, OperationContext operationContext, CancellationToken cancellationToken)
at Microsoft.ADF.PipelineManager.AirflowHandler.ListDAGsAsync(String integrationRuntimeName, String linkedServiceName, String containerName, String folderPath, Boolean copyFolderStructure, SystemProperties systemProperties, CancellationToken cancellationToken) in C:__w1sPipelineManagersrcPipelineManager.CoreJobHandlersAirflowHandlersAirflowHandler.cs:line 165
at Microsoft.ADF.PipelineManager.AirflowController.ListAirflowDAGsAsync(String factoryName, String integrationRuntimeName, ListAirflowDagsRequest request, CancellationToken cancellationToken) in C:__w1sPipelineManagersrcPipelineManager.Core.FXAirflowControllersAirflowController.cs:line 163
Request Information
RequestID:2dd561ad-a01e-0064-643a-bcdb21000000
RequestDate:Sat, 22 Jul 2023 01:18:05 GMT
StatusMessage:This request is not authorized to perform this operation.
ErrorCode:AuthorizationFailure
ErrorMessage:This request is not authorized to perform this operation.
RequestId:2dd561ad-a01e-0064-643a-bcdb21000000
Time:2023-07-22T01:18:05.0775853Z
I’m using a public Integration Runtime for this purpose (not self hosted).
The connexion test to the storage blob is OK.
Aiflow version: 2.4.3
Auth: AAD
I’m trying to attatch the DAGS python files to Airflow from Azure Storage but I got an exception.
2
Answers
I was able to import DAGs from a Blob Storage to a managed Airflow instance using Azure Data Factory, with success following the steps from official documentation
However from your error I assume that you also use Azure Functions or ADF Pipeline to access storage using via managed identity
Can you try and add the ADF MSI to the storage account with the following roles Storage Blob Data Owner / Storage Blob Data Contributor. And let me know how it works?
try following my tutorial:
https://techcommunity.microsoft.com/t5/fasttrack-for-azure/mastering-the-art-orchestrating-adf-with-the-power-of-managed/ba-p/3880515
you need to add your ADF workspace app in AAD just to make it work