within Azure Data Factory I have a table with 2 columns. I want to remove all duplicate rows and only remain one distinct row. Currently: columnA, columnB a1,b1 a1,b1 a2,b2 a3,b3 a3,b3 a3,b4 After the transformation: columnA, columnB a1,b1 a2,b2…
I am using ADF to copy data from Snowflake table (Source) to Azure Blob CSV file (Sink). The Sink dataset has “First row as header” checked. It works well if the source table contains data. However, if the source table…
I'm trying to identify/get the most recently added files and accordingly, transfer the information to an Azure SQL Database table. The problem is that the new files are added to subfolders and I don't know how to search within them.…
I have a pipeline which i'm working on, i'm doing a lookup to check if few files are there, if the folder path is not there i have to perform certain processes, when the folder path is not present, the…
Please help me to write a powershell script to get list of azure data factory triggers list and its scheduling time. Ex: TriggerName,Frequency,ScheduleTime Tr_Copy_Data,Daily,13:30 Thank you in advance. Get list of azure data factory triggers and it’s schedule time for
I'm working in a simple ETL for our budget on MS Azure Data Factory. This pipeline takes a source from an Excel and has a Sink on MS Azure SQL Database. Already done the integration and it works, but i'm…
We are using Azure Data factory as data ingestion service.We have over 100 SHIR at different locations and need to ingest data from all the locations to one centralized DW. So, we need to change SHIR dynamically when we connect…
In a data flow of a Azure Data Factory pipeline I want to preview a specific period of rows in the data preview tab. Whenever I refresh the data preview I see for example data of a year ago, while…
I have configured event trigger for blob storage folder like container/daily_files . Let us say file1.txt and file2.txt will land simultaneously so it will trigger two instances of the same pipeline. I want to control this and execute with only…
I have these messy fields that I need to cleanup in Dataflow. EAN field can have any number of elements separated by ",". But the actual EAN is always the last one. However it seems that there is no way…