I am creating Azure Data Factory pipeline using Python SDK (azure.mgmt.datafactory.models.PipelineResource). I need to convert PipelineResource object to JSON file. Is it possible anyhow? I tried json.loads(pipeline_object) , json.dumps(pipeline_object) but no luck.
I’m using a copy activity in azure synapse pipeline to copy and filter data from containerA/file1.csv to containerB/file2US.csv Similarly I’m using another copy activity to copy and filter data from containerA/file1.csv to containerB/file2IND.csv The same process for different regions. In…
I am looking to use Azure Data Factory to import a number of users from a third party source CSV file into the D365 user table/entity. This is usually straight forward but on this occasion I have a complication. The…
I have an SQL table called FileGroups. I will query for certain columns such as Id, Name and Version. In my Azure Data Factory pipeline, I want to map the column names and row values from the SQL query results…
I'm creating a Data Factory configuration and a pipeline in Python. I successfully added Azure Blob Storage and Azure Key Vault to the linked services, but got stuck on adding (serverless)Azure Synapse Analytics, as simply I was not able to…
I am trying to create an ADF pipeline that does the following: Takes in a csv with 2 columns, eg: Source, Destination test_container/test.txt, test_container/test_subfolder/test.txt Essentially I want to copy/move the filepath from the source directory into the Destination directory (Both…
I want to export my runtime duration of the each pipeline to CSV. How can this be done or achieved? The below is scheduled Pipeline which run to end, i can use the the Export CSV button to get duration…
I am trying to disable Public Network Access of Azure Data Factory through az powershell. As per Microsoft documentation, there is a property called PublicNetworkAccess. So I tried below commands, Import-Module Az.DataFactory Set-AzDataFactoryV2 -PublicNetworkAccess 'Disabled' -Force But it gave me…
I have an excel file as source that needs to be copied into the Azure SQL database using Azure Data Factory. The ADF pipeline needs to copy the rows from the excel source to SQL database only if it is…
I need to get a value returned from a pipeline. Scenario: I am connecting to an API for data. The api needs a token thats generated at the start of the data pipeline. Tokens have a short life and if…