The Copy Activity result doesn’t show the source and sink location (file path or server etc.) Not from the ADF monitor or Log Analytics. It only shows the source and sink type. It is impossible to trace where the data is copied from and to. Does anyone know how to track that information?
Input:
{
"source": {
"type": "ParquetSource",
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "ParquetReadSettings"
}
},
"sink": {
"type": "ParquetSink",
"storeSettings": {
"type": "AzureBlobFSWriteSettings"
},
"formatSettings": {
"type": "ParquetWriteSettings"
}
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": true,
"treatBooleanAsNumber": false
}
}
}
Output
{
"dataRead": 1362,
"dataWritten": 1362,
"filesRead": 1,
"filesWritten": 1,
"sourcePeakConnections": 3,
"sinkPeakConnections": 2,
"copyDuration": 97,
"throughput": 0.227,
"errors": [],
"effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (Canada Central)",
"usedDataIntegrationUnits": 4,
"billingReference": {
"activityType": "DataMovement",
"billableDuration": [
{
"meterType": "AzureIR",
"duration": 0.06666666666666667,
"unit": "DIUHours"
}
],
"totalBillableDuration": [
{
"meterType": "AzureIR",
"duration": 0.06666666666666667,
"unit": "DIUHours"
}
]
},
"usedParallelCopies": 1,
"executionDetails": [
{
"source": {
"type": "AzureBlobFS",
"region": "Canada Central"
},
"sink": {
"type": "AzureBlobFS",
"region": "Canada Central"
},
"status": "Succeeded",
"start": "10/3/2023, 2:33:57 PM",
"duration": 97,
"usedDataIntegrationUnits": 4,
"usedParallelCopies": 1,
"profile": {
"queue": {
"status": "Completed",
"duration": 89
},
"transfer": {
"status": "Completed",
"duration": 6,
"details": {
"listingSource": {
"type": "AzureBlobFS",
"workingDuration": 0
},
"readingFromSource": {
"type": "AzureBlobFS",
"workingDuration": 0
},
"writingToSink": {
"type": "AzureBlobFS",
"workingDuration": 0
}
}
}
},
"detailedDurations": {
"queuingDuration": 89,
"transferDuration": 6
}
}
],
"dataConsistencyVerification": {
"VerificationResult": "NotVerified"
},
"reportLineageToPurview": {
"status": "Succeeded",
"durationInSecond": "1"
},
"durationInQueue": {
"integrationRuntimeQueue": 0
}
}
2
Answers
Just want to update the answer.
Inside the Copy activity, click 'auto-generate' inside 'User Properties'. It will include the source and sink information.
As @Yash Tamakuwala mentioned, enabling logging in the copy activity provides details on the copied file. However, it only displays the copied file name, not the file location, as shown below:
To obtain the file location of the source and sink, use the dataset REST API. Locate the source and sink datasets in the Copy activity JSON. Then, input the required details into the REST API and execute it, as demonstrated below:
The result will appear as follows:
However, if the path is provided dynamically, only the variable will be displayed instead of the file path.