Question 103346 in Azure Question posted in
The official documentation to get you started can be found here.

Azure Data Flow source from output in pipeline

Is it possible to create a Data Flow that gets data from the output of an activity in a Pipeline?

Something like this:
enter image description here

I want to create a pipeline that has an Azure function activity to process some data and then pass that processed data to a Data flow.

But apparently the Data flow only accepts a Dataset as a source.

Tags:

2

Answers


  1. 0

    I dont think this is possible as the data needs to be stored somewhere. My advice would be to put the data somewhere from the function and take it from there to the dataflow. ADF doesnt have the capacity on its own to store data.

    Login or Signup to reply.
  2. 0

    If your Azure Function Activity gives a array of Json, Then you can follow below process:

    Here I have used Lookup Instead of Azure Function which gives Following output:

    enter image description here

    Use this Output in ForEach Activity and generate array for name object like below with append variable activivty inside ForEach:

     [ "Rakesh", "Bhavan", "sai", "kiran", "kowsik", "Rakesh", "kowsik", "arun", "sai", "virat" ]

    Now create a dataflow and with an array parameter. Pass the above array to the Dataflow Parameter from Pipeline:

    enter image description here

    Inside DataFlow take a source with dummy dataset and dummy column,
    Generate the column from the array parameter using derived column expression unfold($arr) like below:

    enter image description here

    Derived Column Result with column:

    enter image description here

    But the above procedure only works for a single array. So, if you have multiple arrays you have to store the Azure function Output to a blob or ADLS and then Get it back to Dataflow as suggested by @Ziya Mert Karakas.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search