Question 104430 in Azure Question posted in
The official documentation to get you started can be found here.

How to check if files present under a data lake directory are empty using Azure Data Factory?

How to check if files present under a data lake directory are empty using Azure Data Factory?

There are multiple files present in the data lake directory and I want to check if these files are empty or not, If files are empty then I want to store the filenames of these empty files in a CSV File.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I have followed this thread Azure Data Factory V2 Check file size for Copy Activity and applied if condition to judge the file size and append variable and set variable to store the file names, Worked for me.


  2. 0

    If you want to check whether the file is empty or not Please follow the below Steps. I tried to reproduce the same in my environment and I got the below results:

    In my storage account, I have two files one is demo123.csv empty and another one is a vm_name3.csv non-empty file.

    enter image description here

    Please follow these Steps:

    Step1: First Create Get metadata with child items

    enter image description here

    Step 2: Add dynamic expression @activity('Get Metadata1').output.childItems on the forEach activity.

    enter image description here

    • Create a dataset parameter
      enter image description here

    Step3 : Inside forEach activity ->use lookup and If condition. Add this dynamic expression on lookup activity : @item().name

    Lookup

    enter image description here

    If condition:

    Using this dynamic expression @equals(activity('Lookup1').output. count,0).You will know whether a file is empty or not.

    enter image description here

    • Then add append variable inside True conduction.
      enter image description here

    • I created two variables with array type.

    enter image description here

    • Add this dynamic expression: @item().name on append variable
      enter image description here

    Step4:

    Add set variable to the forEach activity:
    enter image description here

    After successful execution of pipeline. I got empty file name:

    enter image description here

    If you want to store the filenames of these empty files in a CSV File .Then follow this SO thread by Aswin

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search