One pipeline loads data from an API to ADLS gen2. I have fixed the file's name so that it will replace the file in every run of the pipeline. I am looking to get the number of rows available in…
I am setting up an Azure Data Factory that links to a Postgres SQL server that is hosted elsewhere. Several of our pipelines are failing to copy data from their source Postgres tables into their Azure SQL server sink tables.…
I'm trying to ingest data from the open-source, public Yahoo Finance API using Azure Data Factory. The endpoint I'm testing is https://query2.finance.yahoo.com/v8/finance/chart/GOLD. I am able to ingest the data but I'm coming up with an issue when trying to transform…
I'm trying to copy Azure DevOps board's Analytic shared view data into Fabric Lakehouse. It is a custom view created using Workitem fields and have shared view setting (All users on your project can see this view and use it…
How do you query the content in JSON files using Azure Data Factory? I want to filter any JSON files that contain criteria "code":"B" in markings>document>code and "date":"2020-01-01" in metadata>publication date>date. If there is a match, transfers the json files…
Inside a Azure synapse pipeline, I need to find and compare the row count of the CSV file with summary file stored in Azure Blob storage. If the comparison fails then pipeline will stop, if the counts match then can…
I am building a pipeline in Fabric and I would like to receive an email every time an activity within that dataflow is completed. The problem is that I cannot figure out how to set a variable equal to the…
I only want the Log Error activity to be executed only when any of the set variable activity executes ie: Only when there is an error.But it always executes Like this Not sure what is wrong here.How can i configure…
I'm using a stored procedure to query SQL database tables and the result is a JSON string. I would like to store the string in an Azure storage blob but can't seem to figure out how to just store the…
We have few REST linkedServices in ADF which are making API calls to get some data. The problem is, they only seems to work in GIT mode or preview but not on the Live Data Factory and the JSON looks…