Question 247159 in Azure Question posted in
The official documentation to get you started can be found here.

Azure – Get number of rows copied in ADF, can't see rowsCopied

I have a copy activity to copy a csv file from one container in the data lake to another container in the data lake. Source and sink datasets are binary format type with identical settings. I can’t see rowsCopied in the output.

I want to get number of records copied to the sink to log this into an audit table.

How do I achieve this?

Tags:

2

Answers


  1. 0

    In case of file movements especially Binary formats, I am not sure you would get the rows copied data via Copy activity logs.
    You might have to use a lookup activity and iteration activity to get the row count explicitly or dataflows to get the count.
    https://learn.microsoft.com/en-us/answers/questions/35029/how-to-get-row-counts-from-files-stored-in-azure-s

    Login or Signup to reply.
  2. 0

    AFAIK, for binary files are the number of rows copied are not showed in Copy activity output and it is also not supported in DataFlow

    As your concern is to log number of copied rows into an audit table you need to use storage event trigger which will trigger the pipeline which get the number of copied rows from just coped CSV file.

    • Create a pipeline with Pipeline parameters Folderpath and Filename.
      enter image description here
    • Create a Storage Event Trigger For blob created in particular sink Container.
      enter image description here
    • For this trigger Add trigger run parameter @triggerBody().folderPath and @triggerBody().fileName respectively for Folderpath and Filename.
      enter image description here
    • Now take Lookup activity to pipeline and add ADLS dataset with Delimited text file format with Folderpath and Filename parameters.
      — Dataset for lookup:
      enter image description here
      — Source setting for this:
      enter image description here

    Whenever CSV file is added to particular folder It will trigger this pipeline and you can get the no of rows copied through the lookup output count

    enter image description here

    you can access it using. @activity('Lookup1').output.count in further activities.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search