I am making a rest call using a copy activity to write data to a datalake storage. Source is a rest dataset and sink is a json. If I use a http binary dataset it works fine, but then I can not perform pagination in an easy way. I am getting the following error:
{
"errorCode": "2200",
"message": "Failure happened on 'Source' side. ErrorCode=JsonInvalidDataFormat,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when deserializing source JSON file ''. Check if the data is in valid JSON object format.,Source=Microsoft.DataTransfer.ClientLibrary,'",
"failureType": "UserError",
"target": "ingest json to landing",
"details": []
}
If I only preview the data in adf I am getting the following error:
Error code
21155
Details
Error occurred when deserializing source JSON file ”. Check if the data is in valid JSON object format.
If I make the same call with postman, no issues, but the body comes back as text and looks as follows:
enter image description here
[
{
"Data":1561,
"Number":156,
"ID":1565,
"TypeID":15,
"Type":"sdfg",
"Comments":"jbgijdresbgiesugbiiergburesgbiergb breijfberiugbi iuergiuer erguiergeriu erreuguierh guierhger eu u geir er erug iug eruhiuguiergiuguer er ug e eruhgeuirge erug erugeurhgeueruig .nTipsn1.trjgnrig reigeirg eirghi : nall erjgeriugher ergierig I er gheh erh eirghherigerg ger er ghiergier eh egrigerig eg . n2. Ewgheri ughergh erh r erhgerh:ergjoi ihg iergierg iererigiergi er gieriger er gier iergpai g aei[g rhe agieg i[e I erg hhg rggergieerig ei gerh ergio ehaigneigrdrg dosg .n3. Jsbdiujasbfib:npay erkgierogh erigerho ig er erh oegiuerghe g e ir oego gerghierghe ge rgerihgeri gegh eregh.n"
},
{
"Data":1561,
"Number":156,
"ID":1565,
"TypeID":15,
"Type":"sdfg",
"Comments":"jbgijdresbgiesugbiiergburesgbiergb breijfberiugbi iuergiuer erguiergeriu erreuguierh guierhger eu u geir er erug iug eruhiuguiergiuguer er ug e eruhgeuirge erug erugeurhgeueruig .nTipsn1.trjgnrig reigeirg eirghi : nall erjgeriugher ergierig I er gheh erh eirghherigerg ger er ghiergier eh egrigerig eg . n2. Ewgheri ughergh erh r erhgerh:ergjoi ihg iergierg iererigiergi er gieriger er gier iergpai g aei[g rhe agieg i[e I erg hhg rggergieerig ei gerh ergio ehaigneigrdrg dosg .n3. Jsbdiujasbfib:npay erkgierogh erigerho ig er erh oegiuerghe g e ir oego gerghierghe ge rgerihgeri gegh eregh.n"
}
]
This is how the copy activity is configured:
Does anybody know if a rest dataset supports array of jsonlines?
2
Answers
This is what i did , I took the JSON which you had pasted above . I used that JSON as source and check how ADF handled that .When i tried to preview that data in ADF ( configured the JSON as source) , it failed . I think that the character "e" is for some reason breaking the JSON . I have no idea at this time , why thats happening
Note: When you are using a binary dataset in copy activity. you only copy from binary dataset to binary dataset otherwise it will throw an error.
Request method : GET
Add additional header :
Authorization : Bearer < Token >If you want multiple responses into a single file it will break the JSON format and lead to invalid JSON. Please change the file pattern to an Array of object or Set of object.
Output:
For more details refer this SO thread and MS Document