Azure Question posted in
The official documentation to get you started can be found here.

Azure – can get metadata activity fetch data recursively at ADF pipeline?

i am currently mainly doing the practicing in ADF pipeline and I realised if i use getmetadata activity and select child items it won’t go deep inside of the folder of the folder, which is not doing recursively, is anyway can achieve it? i found if directly using copy activity and select wildcard path can active recursive and allow all the files no matter in which folder can copy to the destination, but if i need additional function like filter by last modify date or filter by other condition I still need use get metadata and filter activity

Tags:

2

Answers


  1. 0

    When you get metadata from file stores and configure modifiedDatetimeStart or modifiedDatetimeEnd, the childItems in the output includes only files in the specified path that have a last modified time within the specified range. Items in subfolders are not included.

    The output of getmetadata includes only files in the specified path to get nested values you need to iterate on output directories of getmetadata values. using foreach loop this way you can go up to 1 level deep.

    enter image description here

    @Richard Swinbank here Get Metadata recursively in Azure Data Factory document discussed the Workaround for same situation.

    An alternative to attempting a direct recursive traversal is to take an iterative approach, using a queue implemented in ADF as an Array variable. You can follow above documentation to create this iterative approach.

    Login or Signup to reply.
  2. 0

    GetMetaData activity, will not allow us to get child items recursively. We can consider doning loop using ForEach activity. And inside loop, we can consider using again GetMetaData activity.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search