Question 171712 in Azure Question posted in
The official documentation to get you started can be found here.

Does Azure-ADF Delete Activity delete Folders?

I have a pipeline to delete some folders from ADLS. My folder structure is as below.

/raw/MainFolder/SubfolderA/20230430/File.parquet
/raw/MainFolder/SubfolderA/20230415/File.parquet
/raw/MainFolder/SubfolderA/20230410/File.parquet

/raw/MainFolder/SubfolderB/20230430/File.parquet
/raw/MainFolder/SubfolderB/20230420/File.parquet
/raw/MainFolder/SubfolderB/20230405/File.parquet

The pipeline is currently deleting the File.parquet under the date folder. But I need it to delete the whole folder named with date (20230430).

Currently the pipeline is deleting the parquet files when they exist and erroring out when the folder is empty.
I am not passing the parquet file name to my pipeline. (Just the folder path)
I have recursively enabled in the delete activity.

Error: Failed to execute delete activity with data source ‘AzureBlobStorage’ and error ‘The required Blob is missing. Folder path: raw/MainFolder/SubfolderA/20230430/.’.

How do I get the delete activity to delete the folder itself and not just the files in it?

I am very new to Azure data factory.
Appreciate your help.

Pipeline is deleting the files but not the folders

Tags:

2

Answers


  1. 0

    • Kindly make sure that the logging account is not pointing to the same folder which you are trying to delete. As while capturing the log details, delete activity will use the same folder which you have specified in the logging settings.

    • Here is the gif where I am using different folder for logging purpose and the folder which I am intended to delete is getting deleted along with the parquet file within it.

      dev/person — I am trying to delete person folder in the below gif:
      enter image description here

    Login or Signup to reply.
  2. 0

    Error: Failed to execute delete activity with data source ‘AzureBlobStorage’ and error ‘The required Blob is missing. Folder path: raw/MainFolder/SubfolderA/20230430/.’.

    The above error will cause because of For ADLS path you are using blob storage linked service.

    In blob storage you can’t delete empty directory/folder .

    I also got the same error when i used blob storage linked service.

    enter image description here

    To resolve this use ADLS linked service, and to delete particular folder from ADLS irrespective of file contain in it.

    • First take a delete activity and create a source dataset with ADLS linked service for it with just folder name.
      enter image description here
      Source dataset settings: folder name is MainFolder/SubfolderA/20230430
      enter image description here

    This will delete respective folder.

    before deleting folder:
    enter image description here
    After deleting folder 20230430:
    enter image description here

    Dataset.json

    {
    "name": "Binary1",
    "properties": {
        "linkedServiceName": {
            "referenceName": "AzureDataLakeStorage1",
            "type": "LinkedServiceReference"
        },
        "annotations": [],
        "type": "Binary",
        "typeProperties": {
            "location": {
                "type": "AzureBlobFSLocation",
                "folderPath": "MainFolder/SubfolderA/20230430",
                "fileSystem": "raw"
            }
        }
    }
}

    Pipeline.json

    {
    "name": "pipeline1",
    "properties": {
        "activities": [
            {
                "name": "Delete1",
                "type": "Delete",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "dataset": {
                        "referenceName": "Binary1",
                        "type": "DatasetReference"
                    },
                    "logStorageSettings": {
                        "linkedServiceName": {
                            "referenceName": "AzureDataLakeStorage1",
                            "type": "LinkedServiceReference"
                        },
                        "path": "raw/logs"
                    },
                    "enableLogging": true,
                    "storeSettings": {
                        "type": "AzureBlobFSReadSettings",
                        "recursive": true,
                        "enablePartitionDiscovery": false
                    }
                }
            }
        ],
        "annotations": []
    }
}
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search