Question 267571 in Azure Question posted in
The official documentation to get you started can be found here.

Azure – ADF single event trigger for multiple files

I have configured event trigger for blob storage folder like container/daily_files . Let us say file1.txt and file2.txt will land simultaneously so it will trigger two instances of the same pipeline. I want to control this and execute with only single instance of ADF pipeline. Is there any way we can achieve this ?

Tags:

2

Answers


  1. 0

    Unfortunately there is no direct way.
    You would have to either :

    1. Make the parallel execution of the pipeline 1, so that even if the pipeline is triggered twice, the execution would be sequential. And then process all files in initial execution, archive them and in 2nd execution skip the execution.
      somewhat similar :
      https://datasharkx.wordpress.com/2022/09/24/event-trigger-of-synapse-azure-data-factory-pipeline-on-arrival-of-nth-file-in-azure-blob-storage/

    2. The blob trigger can be on a pipeline which can in turn trigger your main pipeline based on validating the file count

    Login or Signup to reply.
  2. 0

    AFAIK there is no way to control pipeline run once it is got triggered simultaneously.

    You can set pipeline Concurrency to 1. It will be act like it would start pipeline run for the first trigger if pipeline got triggered in between the first run it will add second run in queue.

    enter image description here

    Note – Please be aware that there is a limit on the size of the queue. Using pipeline concurrency of 1, for example, might result in one pipeline run being in progress and 100 queue entries; if this happens and the pipeline continues to trigger, error 429 will occur.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search