Within an Azure Data Factory Pipeline I am attempting to remove an empty directory.
The files within the directory were removed by a previous pipelines’ iterative operation thus leaving an empty directory to be removed.
The directory is a sub-folder: The hierarchy being:
container / top-level-folder (always present) / directory – dynamically created – the result of an unzip operation.
I have defined a specific dataset that points to
container / @concat(‘top-level-folder/’,dataset().dataset_folder)
where ‘dataset_folder’ is the only parameter.
The Delete Activity is configured like this:
On running the pipeline it errors with this error:
Failed to execute delete activity with data source ‘AzureBlobStorage’ and error ‘The required Blob is missing. Folder path: container/top level directory/Directory to be removed/.’. For details, please reference log file here:
The log is an empty spreadsheet.
What am I missing from either the dataset or delete activity?
2
Answers
I used an Azure function suggested here: Delete folder using Microsoft.WindowsAzure.Storage.Blob : blockBlob.DeleteAsync();
to perform the action.
In azure blob storage, when all the contents inside a folder are deleted, then the folder would automatically get deleted.
This is because the folder is deleted as soon as the files in the folders are deleted. Only when using azure data lake storage, you will have to specifically delete the folder as well.
Since the requirement is to delete the folder in azure blob storage, you can simply remove the delete activity that you are using to remove the folder.