Question 105046 in Azure Question posted in
The official documentation to get you started can be found here.

Azure Data Factory find which pipeline in a sequence failed

I have a pipeline (PL_Main) containing a set of Execute pipeline activities in sequence, as the below image depicts:
enter image description here
Now, I was trying to create an activity (executed after PL_Main completion) that allowed me to understand if any of the ‘childs’ Execute Pipeline activities (PL_1, PL_2, PL3.1, PL_3.2) failed.

However, according my understanding thee pipelines output contains only the RunId and PipelineName. Is there a way that I can find what were the pipelines in my execution chain that failed (or at least the 1st one failling) and store their RunId in as a parameter for my next activity (New Activity) ?

Thanks in advance to everybody!

Tags:

2

Answers


  1. 0

    Assuming you need to pass the pipelinerunid of the failed activity within Main pipeline, you need to deign the pipeline in below way :

    enter image description here

    JSON:

    {
"name": "pipeline8",
"properties": {
    "activities": [
        {
            "name": "Execute Pipeline1",
            "type": "ExecutePipeline",
            "dependsOn": [],
            "userProperties": [],
            "typeProperties": {
                "pipeline": {
                    "referenceName": "pipeline7",
                    "type": "PipelineReference"
                },
                "waitOnCompletion": true
            }
        },
        {
            "name": "Execute Pipeline2",
            "type": "ExecutePipeline",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline1",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "pipeline": {
                    "referenceName": "pipeline9",
                    "type": "PipelineReference"
                },
                "waitOnCompletion": true
            }
        },
        {
            "name": "Execute Pipeline3",
            "type": "ExecutePipeline",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline2",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "pipeline": {
                    "referenceName": "pipeline10",
                    "type": "PipelineReference"
                },
                "waitOnCompletion": true
            }
        },
        {
            "name": "Execute Pipeline4",
            "type": "ExecutePipeline",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline2",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "pipeline": {
                    "referenceName": "pipeline11",
                    "type": "PipelineReference"
                },
                "waitOnCompletion": true
            }
        },
        {
            "name": "Fail1",
            "type": "Fail",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline1",
                    "dependencyConditions": [
                        "Failed"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "message": {
                    "value": "@activity('Execute Pipeline1').output.pipelineRunId",
                    "type": "Expression"
                },
                "errorCode": "1"
            }
        },
        {
            "name": "Fail2",
            "type": "Fail",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline2",
                    "dependencyConditions": [
                        "Failed"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "message": {
                    "value": "@activity('Execute Pipeline2').output.pipelineRunId",
                    "type": "Expression"
                },
                "errorCode": "2"
            }
        },
        {
            "name": "Fail3",
            "type": "Fail",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline3",
                    "dependencyConditions": [
                        "Failed"
                    ]
                },
                {
                    "activity": "Fail5",
                    "dependencyConditions": [
                        "Skipped"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "message": "@activity('Execute Pipeline3').output.pipelineRunId",
                "errorCode": "1"
            }
        },
        {
            "name": "Fail4",
            "type": "Fail",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline4",
                    "dependencyConditions": [
                        "Failed"
                    ]
                },
                {
                    "activity": "Fail5",
                    "dependencyConditions": [
                        "Skipped"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "message": "@activity('Execute Pipeline4').output.pipelineRunId",
                "errorCode": "4"
            }
        },
        {
            "name": "Fail5",
            "type": "Fail",
            "dependsOn": [
                {
                    "activity": "Execute Pipeline3",
                    "dependencyConditions": [
                        "Failed"
                    ]
                },
                {
                    "activity": "Execute Pipeline4",
                    "dependencyConditions": [
                        "Failed"
                    ]
                }
            ],
            "userProperties": [],
            "typeProperties": {
                "message": {
                    "value": "@concat(activity('Execute Pipeline3').output.pipelineRunId,';',activity('Execute Pipeline4').output.pipelineRunId)",
                    "type": "Expression"
                },
                "errorCode": "5"
            }
        }
    ],
    "variables": {
        "test": {
            "type": "String"
        },
        "spt": {
            "type": "Array"
        },
        "final": {
            "type": "String"
        }
    },
    "annotations": []
}

    }

    Reference blog : https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory/

    Note: the fail activities capture the pipelinerunid

    Finaly to capture the ID:
    enter image description here

    @activity('Execute Pipeline1').error.message
    Login or Signup to reply.
  2. 0

    You can try this approach also, to get the failed pipeline run id.

    For this name the Execute pipeline activities name same as child pipeline names.

    First get the Failed Execute pipeline activity name (Pipeline name) in the main pipeline using the error message.

    Error message:

    enter image description here

    Use set variable to get this.

    enter image description here

    Get the pipeline name from the error message.

    use the below expression for that.

    @substring(variables('error'), add(indexOf(variables('error'),'target'),7), sub(indexOf(variables('error'),' failed'),add(indexOf(variables('error'),'target'),7)))

    enter image description here

    You can see that we got the pipeline name in the variable

    enter image description here

    Now get all the pipeline runs from the Pipeline Runs – Query By Factory command.

    POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/queryPipelineRuns?api-version=2018-06-01

    Use this in web activity.

    This will give you all pipeline runs details with names and run ids Json which you can see in the above documentation.

    Now compare our pipeline name from variable in this JSON and get that particular run id which is our required failed pipeline run id. Use ForEach activity or filter for that.

    Reference:

    Check this SO thread for reference by KarthikBhyresh-MT

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search