Question 349136 in Azure Question posted in
The official documentation to get you started can be found here.

Azure Data Factory: Validate the existence of more than one files before performing the copy operation

I have a use case: there are four files existing in different directories. I want to copy all four files only if they are all present. How do I validate the existence of all four files together? I have created a config file from which I am reading the path values dynamically, but I do not know how to validate the existence of all the files together. I am following the below approach to create a validation pipeline.

  1. Lookup Activity to Extract Path Details:

    • Configure a Lookup activity to read the path details from your configuration file (e.g., a JSON file stored in Azure Blob Storage).
    • Set up the Lookup activity to retrieve the paths of the four files located in different Azure Blob directories.

  2. Foreach Activity:

    • Use a Foreach activity to iterate over the paths extracted from the configuration file.
    • Configure the Foreach activity to loop through the array of paths obtained from the Lookup activity.
    • Set the items property of the Foreach activity to the array containing the paths.

  3. Get Metadata Activity:

    • Within the Foreach loop, include a Get Metadata activity to check whether each file exists.
    • Configure the Get Metadata activity to use the dynamically changing file paths provided by the Foreach activity.
    • Set the dataset property of the Get Metadata activity to the dataset representing the Azure Blob Storage where your files are located.
    • Use an expression to dynamically set the file path for each iteration of the Foreach loop.

  4. If Condition:

    • After the Get Metadata activity, add an If condition activity to check if the file exists or not.
    • Configure the If condition activity to evaluate whether the file exists based on the output of the Get Metadata activity.
    • If the file exists, increment a counter variable. You can achieve this by using a variable and an expression within the If condition activity.
    • @equals(activity(‘Get Metadata Activity’).output.exists, true)

  5. Set Variable : inside the if condition (step 4)

    • After each iteration increment the counter value.
    • @add(variables(‘FileCounter’), 1)

  6. Completion Condition:

    • After the Foreach loop, add an activity to check if the counter variable equals 4. This condition ensures that all four files exist before triggering the next pipeline.
    • Use an expression to evaluate whether the counter variable equals 4.
    • @equals(variables(‘FileCounter’), 4)

  7. Trigger Another Pipeline:

    • If the completion condition is met (i.e., all four files exist), trigger another pipeline using the appropriate activity.

I am getting the error at stage 5. Here after adding the logic to increment the counter I am getting the error.
The expression contains self referencing variable. A variable cannot reference itself in the expression.
so over all I am struggling at stage 5 and 6.
Note: I have declared the FileCounter variable at pipeline level.

Tags:

2

Answers


  1. 0

    You can follow the below approach :

    1. Use Get meta data activity and get the list of all files present within the folder as Child items property
    2. Use a lookup activity to get the list of expected files
    3. use filter activity to get the list of matching files and in case if that count is equivalent to overall file count, then proceed ahead .

    Somewhat similar thread for reference:

    how to check and compare the file names that are inside a folder (Datalake) using ADF

    Login or Signup to reply.
  2. 0

    As you have a config file consisting of file paths, use that in lookup activity. Follow the below approach to achieve your requirement.

    For sample I have taken the config.csv file like below.

    paths
con3/folder1/Table2.csv
con3/folder2/sample_blank.csv
con3/folder3/Table2.csv
con3/folder4/Table2.csv

    enter image description here

    Create two variables flag of Boolean type and filepath of String type like below and set a default value true to flag variable.

    enter image description here

    After lookup, use for-each activity and pass the lookup output array to it.

    enter image description here

    Create a dataset and in that, create a string parameter. Use that in the file name of the dataset like below.

    enter image description here

    Use this dataset in the Get meta data activity inside for-each. Give the @item().paths to the paramater and use the property Exists here.

    enter image description here

    After this, use if activity and check the expression @activity('Get Metadata1').output.exists in that.

    In the False activities of if, update the variables flag to false and filepath to @item().pathsusing set variable activities.

    enter image description here

    After the debug of the pipeline, at the end of the pipeline, if any file does not exist, the flag variable value will be set to false and filepath variable will consists of the respective file path as well. The flag will stay as true if all files exists.

    enter image description here

    After for-each activity, use an if activity and check the flag variable in it. If True, go ahead with your copy activity operation in True activities of if. If False, use the respective activities to send the message to your user in the False activities of if.

    The expression contains self referencing variable. A variable cannot reference itself in the expression.

    Coming to your approach, as per the Documentation, self-referencing variables is not supported in ADF. You need to use a temporary variable to update the original variable. Go through the method suggested in the documentation for better understanding.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search