Question 105694 in Azure Question posted in
The official documentation to get you started can be found here.

Unable to read Parquet files(Created from Databricks Job/Notebook) from Azure Data Factory

I am getting error while I am reading Parquet file created by Databricks on ADLS. While I read these files using Databricks it works perfectly fine and I am able to read and write data into these files from Databricks. However with DataFactory it is giving below error.

Error: Parquet file contained column 'txn', which is of a non-primitive, unsupported type.

However there is no txn column create by me from Databricks.

Tags:

2

Answers


  1. 0

    This error mainly happens because of unsupported data type. When you pass to the column in parquet file make sure you are using the supported datatype.

    Supported datatype mapping for parquet file refer this Microsoft document .

    Login or Signup to reply.
  2. 0

    When you have complex data types in your parquet source files in ADF, you need to use a data flow without a dataset schema. Then you can work with structs, maps, arrays, etc: https://www.youtube.com/watch?v=Wk0C76wnSDE

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search