Question 103546 in Azure Question posted in
The official documentation to get you started can be found here.

Additional column throwing validation issue with Azure SQL data sink in Azure Data Factory

Validation Error

I’ve got this weird issue where validation fails on ‘additional columns’ for my data sink to Azure SQL coming from a blob storage source in the Azure Data Factory GUI. No matter how many times we recreate the dataset (or specify another dataset, new) we can’t get past this validation issue.

The irony of this is we deploy these pipelines from code and when we run them, we get no errors at all. This issue we have had just made life really difficult developing pipelines further as we have to do everything by code. We cant use the pipepline publish option.

Here are some screen grabs for you of the pipeline so you can see the flow.

Pipeline

Inside copyCustomer.

Source

Mapping

Sink

Any ideas on how to fix this validation would be greatly appreciated.

For what it’s worth, we have recreated the dataset multiple times (clone and new) to avoid any issue with the dataset model not being the latest as per what’s documented here https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview#add-additional-columns-during-copy

Sometimes by setting the table in sink to autocreate has shown the validation to be ‘fixed’ but then when we go to publish it errors out again.

Tags:

2

Answers


  1. 0

    When your Azure SQL dataset was created long time before and is still utilizing an outdated dataset model that Additional Columns do not support, this is expected behavior.

    As per official Microsoft documentation

    To resolve this issue, you can just follow the error message to create a new Azure SQL dataset and use this as copy sink.
    enter image description here

    I followed error message and created new data set and it is working fine for me.

    Source:
    enter image description here

    Mapping:
    enter image description here

    Sink:
    enter image description here

    Output:
    enter image description here

    Login or Signup to reply.
  2. 0

    I suspect here, your dataset of Sink type is incorrect. I reproduced,
    same at my end. Its working fine. Kindly make sure you create a sink dataset type with Azure SQL database type connector only.
    Please check below screenshots from my implementation.
    enter image description here
    enter image description here
    If still it helps, feel free to share your sink dataset type connector details along with screenshots.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search