To start, it seems like pagination works differently when using a Copy data command vs. using a Data Flow to do the same thing, but adding a flatten transformation inbetween the ingest and sink. When using the Copy data command,…
I have a file which is present as a unstructured txt in ADLS. I want to read the entire file content as text and pass it to a Stored Procedure in Azure data factory. Can this be done through Azure…
I'm trying to copy data from Salesforce to Snowflake using ADF avoiding use SAS Token, because of that I going to use integration storage on Snowflake. I'm trying to follow this example: https://medium.com/snowflake/seamless-migration-to-snowflake-using-adf-script-activity-schema-detection-25475ea86a09. I already create the integration storage, format…
This is my input file - It is just a sample It will have more than 20k rows {"A":12345,"B":"2023-12-31 00:00:00","C":5510,"D":20231231,"E":20240103,"F":20231231,"G":1577158,"H":63217,"I":null,"J":"Company123","K":"SupplierXYZ","L":"987","M":"Vendor ABC","N":"Invoice789","O":"2023-12-31T00:00:00","P":"2024-01-03T00:00:00","Q":"2023033344","R":"7559","S":"Dept123","T":"ModelXYZ","U":7179.0,"V":3,"W":"3769686.0","X":"TypeJ","Y":7179.0,"Z":null,"AA":"NOK","AB":"202312.0","AC":"EXTERNAL","AD":null,"AE":"Counterparty999","AF":"2023-12-31T00:00:00","AG":null,"AH":null,"AI":"66c6c4cc31107992a4079f3653a8a86930a2675d8663c5ab2ec4df23b0f6f3b0","AJ":"2024-02-13T12:58:02.597","AK":"2024-01-03T08:31:16.425292"} {"A":54321,"B":"2024-01-15 08:00:00","C":7890,"D":20240115,"E":20240120,"F":20240115,"G":2468135,"H":951753,"I":null,"J":"Company456","K":"Supplier789","L":"123456789","M":"Vendor XYZ","N":"Invoice123","O":"2024-01-15T08:00:00","P":"2024-01-20T08:00:00","Q":"2024011515","R":"1234","S":"Dept456","T":"ModelABC","U":9876.0,"V":6,"W":"9876543.0","X":"TypeK","Y":9876.0,"Z":null,"AA":"USD","AB":"202401.0","AC":"INTERNAL","AD":null,"AE":"Counterparty888","AF":"2024-01-15T08:00:00","AG":null,"AH":null,"AI":"a1b2c3d4e5f6g7h8i9","AJ":"2024-02-15T08:00:00.123","AK":"2024-01-20T08:00:00.987654"} I want the output as { "dataRepositoryId": "fixedvalue", "importName": "uuid, "data": [ {"A":12345,"B":"2023-12-31 00:00:00","C":5510,"D":20231231,"E":20240103,"F":20231231,"G":1577158,"H":63217,"I":null,"J":"Company123","K":"SupplierXYZ","L":"987","M":"Vendor ABC","N":"Invoice789","O":"2023-12-31T00:00:00","P":"2024-01-03T00:00:00","Q":"2023033344","R":"7559","S":"Dept123","T":"ModelXYZ","U":7179.0,"V":3,"W":"3769686.0","X":"TypeJ","Y":7179.0,"Z":null,"AA":"NOK","AB":"202312.0","AC":"EXTERNAL","AD":null,"AE":"Counterparty999","AF":"2023-12-31T00:00:00","AG":null,"AH":null,"AI":"66c6c4cc31107992a4079f3653a8a86930a2675d8663c5ab2ec4df23b0f6f3b0","AJ":"2024-02-13T12:58:02.597","AK":"2024-01-03T08:31:16.425292"}, {"A":54321,"B":"2024-01-15…
I have a dataset setup in my ADF solution which points to a blob container. There are 25 CSV files sitting in this container. I have a dataflow setup in ADF which currently points to each .CSV file respectfully. For…
We have web activity and it always return below string with value change (value is dynamic everyday). We would need to extract key value. Please advise how to achieve this in Azure Data Factory activities. Set Activity return value :…
I have a database delta file (.sql) and it contains sql statements such as Insets, Delete, Update. The database is setup with the tables and data, I am simply trying to execute these updates I am trying to find a…
I have an ADF package setup in Azure. There are 25 files that i am connecting to using the dataset setup in ADF. There are 25 connections, 1 connection to each .CSV file (the .CSV files will include a date…
I'm facing an issue while passing boolean values within a JSON template in Azure Data Factory (ADF). My scenario involves transmitting a boolean variable from ADF to a Logic App. However, I encounter errors depending on how I handle the…
I have a use case: there are four files existing in different directories. I want to copy all four files only if they are all present. How do I validate the existence of all four files together? I have created…