Question 232791 in Azure Question posted in
The official documentation to get you started can be found here.

Azure – Data Factory succeeds but no data at target

I am running an incremental process for multiple tables where I am using old watermark, new watermark, copy activity, and a stored procedure to update the watermark value. One of my table succeeds when I run the pipeline, for eg I can see total rows copied as 900 whic matches source SQl but when I check in the Database I can see only 70 rows getting reflected, I am not sure why I am not able to see all the rows at database level. Is this some kind of bug in ADF. Need some advise and suggestions on my situation

Thank You

Tags:

2

Answers


  1. 0

    Its not the first time I am seeing this honestly. Can you implement data consistency verification in copy activity? And what is your fault tolerance?

    https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-data-consistency

    enter image description here

    Some rows may be skipped, in this case for example ADF will still show you like they are "written" because logically they are, but physically not, because of incompatible rows etc. which are skipped. In other words, skipped rows are counted as "written" in ADF terminology. Should check that as well

    Login or Signup to reply.
  2. 0

    This happens the rows are rejected at the sink.
    Do the following to confirm and fix the error.

    1. check what setting is enabled under copy activity –> settings –> fault tolerance. if it is skip incompatiable rows, the job won’t fail even if the rows are rejected.
    2. Under copy activity settings tick enable logging and output the rejected rows to your storage account.
    3. The reason for skipping the rows can be identified from this logs.
    4. Make necessary changes to the sink table and re run the job.
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search