Question 105668 in Azure Question posted in
The official documentation to get you started can be found here.

One source to many sinks azure synapse pipeline?

I’m using a copy activity in azure synapse pipeline to copy and filter data from
containerA/file1.csv to containerB/file2US.csv

Similarly I’m using another copy activity to copy and filter data from containerA/file1.csv to containerB/file2IND.csv

The same process for different regions. In every activity I add a where clause to filter the data and copy it into region specific files.

It feels pretty redundant to do this way. Is there any way where I can conditionally check each row and copy it to a different sink based on the region value?
What I’m trying to achieve is a SINGLE ACTIVITY that can select the correct sink based on a condition each row maps to.

Tags:

2

Answers


  1. 0

    The activity you are looking for is called Data Flows. You will use the Conditional Split transformation with as many sinks as you require to achieve this use case.

    Login or Signup to reply.
  2. 0

    I would approach this using a For Each activity which runs in parallel and a parameterised Copy activity. You can use an array parameter to list the regions you want to loop through. Here’s an example with continents:

    ["Africa","Antarctica","Asia","Australia","Europe","North America","South America"]

    Set up your pipeline like this:

    logical synapse pipeline

    Use the Query in the Sink and and parameterise it with the Add dynamic content button:

    Copy activity

    Alternately use a Stored Proc. Parameterise the Sink using a dataset parameter. This will give you control of the output filename and location.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search