Me and my team are using Azure Synapse Analytics to ingest data from a REST API to a Azure Data Lake Storage Gen2, in order to create views automatically. The only way we could manage to do this in our…
I'm starting a Databricks notebook from ADF to do some preprocessing tasks. The cluster of this notebook is usually not running and should only run, when the ADF pipeline is running as well. But it takes several minutes for the…
I'm working with a dataset where I need to drop some columns which contain only NULL values. The issue is that the column names are not consistent or similar, and can change with time. I was wondering if there is…
I am working with Synpase Spark Pools in a controlled corporate environment. I have limited permission to query AAD but I can create UAMIs and assign them to Resources. When I access my Synpase workspace I can create a Spark…
I have a JSON file in the following format: { "first_name": "Jane", "last_name": "Doe", "userid": 12345, “profile”: { “annoying_field_name_10”: {“field_id”: 15, “field_name”: “Gender”, “value”: “Female”, “applicable”: 1 }, “annoying_field_name_11”: {“field_id”: 16, “field_name”: “Interests”, “value”: “Baking”, “applicable”: 1 } } }…
I was able to blob test connection and it's successful, but when I attempt to look for the storage path it shows this error. screenshot Full error: Failed to load Blob operation failed for: Blob Storage on container '' and…
If I define a global variable at the management panel level. Is that variable accessible by other concurrently running data factories? I am looking into using a global as a validation flag as it will be available as a return…
In Azure Data Factory I created a pipeline with several copy data activities. As soon as one of them crashes I want that it continues to the next step and that it sends an e-mail with the error. The mail…
I have to copy a file from an HTTP source to Azure Blob Storage (ABS) using a copy activity in Azure Data Factory (ADF). The fully-qualified path to the file has a date-stamp in it, so it keeps changing (e.g.,…
In our ADF application pipeline A is invoking pipeline B. I want to capture the run ID so I used the @pipeline().RunId to capture it. But while executing the two run ID Which is as per design. But I want…