Question 105982 in Azure Question posted in
The official documentation to get you started can be found here.

Azure – Returning a value from a data factory pipeline

I need to get a value returned from a pipeline.

Scenario:
I am connecting to an API for data.
The api needs a token thats generated at the start of the data pipeline.
Tokens have a short life and if they expire need to be renewed.

Rather than have multiple web activities to renew the token all over the place, i want just one pipeline that gets the token and returns it. the pipeline can be called where needed.
As the web call to get a token takes a lot of parameters, if i ever have to change it, it would be easier change it in one place than many.

So far i have not seen a way to do this easily in ADF using parameters or variables and would appreciate any expertise.

Thanks!

Tags:

3

Answers


  1. 0

    As @Joel Cochran mentioned in the comments, currently the pipeline will not return any output. You can raise a feature request from the Azure data factory.

    Below are the alternative methods you can use to get the output of child pipeline.

    Method1:

    You can call your pipeline from another pipeline using Execute pipeline activity, but this will not return the actual output of the child pipeline, instead returns the child pipeline RunId.

    Using this pipelineRunID, you can make a call using web activity to get the activity run details which contain the actual output of the pipeline in the response body.

    Example:

    1. Execute child pipeline from parent pipeline using Execute pipeline activity to get the pipeline RunId.

    enter image description here

    1. Make a POST request to get the activity runs using the web activity by adding a dynamic expression in the URL to the get child pipeline RunId to include in the HTTP request.

    URL:

    @concat('https://management.azure.com/subscriptions/{your subscription ID}/resourceGroups/{your resource group name}/providers/Microsoft.DataFactory/factories/{your factory name}/pipelineruns/',string(activity('Execute Pipeline1').output.pipelineRunId),'/queryActivityruns?api-version=2018-06-01')

    enter image description here

    1. You can use the output of the child pipeline from the response body of the web activity in later activities.

    @string(activity('Web1').output.value[0].output)

    Method2:

    You can store the output in a storage or a database and retrieve it again using lookup activity from another pipeline.

    Login or Signup to reply.
  2. 0

    Possibly set a global variable as well.
    Global variables are defined from Manage / Global Parameters menu.

    Login or Signup to reply.
  3. 0

    It’s now possible to return a value from a Pipeline. the Feature is in preview. For more info, have a look at the below article.

    https://praveenkumarsreeram.com/2023/02/15/azure-data-factory-return-value-from-a-child-pipeline-to-a-parent-pipeline/

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search