Question 104398 in Azure Question posted in
The official documentation to get you started can be found here.

How do I pull the last modified file with data flow in azure data factory?

I have files that are uploaded into an onprem folder daily, from there I have a pipeline pulling it to a blob storage container (input), from there I have another pipeline from blob (input) to blob (output), here is were the dataflow is, between those two blobs. Finally, I have output linked to sql. However, I want the blob to blob pipeline to pull only the file that was uploaded that day and run through the dataflow. The way I have it setup, every time the pipeline runs, it doubles my files. I’ve attached images below

[![Blob to Blob Pipeline][1]][1]

Please let me know if there is anything else that would make this more clear
[1]: https://i.stack.imgur.com/24Uky.png

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I was able to solve this by selecting "Delete source files" in dataflow. This way the the first pipeline pulls the new daily report into the input, and when the second pipeline (with the dataflow) pulls the file from input to output, it deletes the file in input, hence not allowing it to duplicate Dataflow


  2. 0

    I want the blob to blob pipeline to pull only the file that was uploaded that day and run through the dataflow.

    To achieve above scenario, you can use Filter by last Modified date by passing the dynamic content as below:

    • @startOfDay(utcnow()) : It will take start of the day for the current timestamp.
    • @utcnow() : It will take current timestamp.

    enter image description here

    Input and Output of Get metadata activity: (Its filtering file for that day only)

    enter image description here

    If the files are multiple for particular day, then you have to use for each activity and pass the output of Get metadata activity to foreach activity as

    @activity('Get Metadata1').output.childItems

    enter image description here

    Then add Dataflow activity in Foreach and create source dataset with filename parameter

    enter image description here

    Give filename parameter which is created as dynamic value in filename
    enter image description here

    And then pass source parameter filename as @item().name
    enter image description here

    It will run dataflow for each file get metadata is returning.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search