Azure Question posted in
The official documentation to get you started can be found here.

Processing .CSV files in Azure ADF (ignoring tailiing dates in filename)

I have an ADF package setup in Azure. There are 25 files that i am connecting to using the dataset setup in ADF. There are 25 connections, 1 connection to each .CSV file (the .CSV files will include a date in the filename of the .CSV i.e. Campus20-02-24.csv. I would like to know in Azure ADF how i can ignore the 20-02-24 and just have a connection to the file as Campus.csv. All of the files sit in a blob container; the ADF solution points to the blob container and processes the file. I would like to ignore the date in the file name and only process .CSV files where the name is like Campus*.csv for example? Is this possible?

I am expecting if the file arrives in the blob container as Campus20-02-24.csv, or Campus21-02-24.csv then i would like to have my dedicated dataset in the ADF package for Campus to process it or identify it as Campus.csv OR perhaps even if its possible to rename the file to Campus.csv before processing because the fixed connection on the data set is looking for Campus.csv, not Campus20-02-24.csv for example.

Tags:

2

Answers


  1. 0

    As your source files are from Blob container, you can use Storage event triggers in ADF.

    Use dataset parameters for the filename of the source dataset and target dataset.

    First, create a string type parameter in the dataset and use in the filename of the source like below.

    enter image description here

    enter image description here

    Similarly, do the same for the target dataset.

    Now, in the pipeline, create a pipeline parameter of string type.

    enter image description here

    Now, take a copy activity with source and target datasets and use this pipeline parameter for the dataset parameters of the source and target datasets.

    enter image description here

    Do the same in the sink of the copy activity as well.

    Now, create a storage event trigger for this pipeline. Give the storage account, container name and in the Blob path begins with option, give Campus word. In the Blob paths ends with option, give .csv like below.

    enter image description here

    Click on continue and at the end of the trigger creation, it will ask the value of the parameter. Here, provide the trigger parameter @triggerBody().fileName to that like below.

    enter image description here

    Click on OK and publish both pipeline and trigger. So, whenever a new file with a name of Campus*.csv uploaded to your Blob container, the trigger will run the pipeline and the file will be copied to the target location like below.

    enter image description here

    This pipeline design is for the daily loading files. To copy the already existing 25 files to the target location, use wild card file path Campus*.csv in the copy activity source like below.

    enter image description here

    Login or Signup to reply.
  2. 0

    You can use Wildcard file path in copy activity if you want to process all the files starting with particular string like ‘campus_2024_01_01.csv’ etc with pattern as ‘campus*’.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search