Question posted in
Some documentation to get you started can be found here.

Error- invalid char between token and delimiter using ConvertRecord in NiFi – Apache

I am using a flow as follows(basically to fetch a file from s3 and then convert few records from the main CSV file n later push it to Elasticsearch) :
GetSQS ->UpdateAtttribute->SplitJson->EvaluateJsonPath->UpdateAttribute->convertRecord-> other processor…

I am able to fetch the file from s3 correctly but the ConvertRecord processor thows error: Invalid char between encapsulated token a delimiter

Please find the ConvertRecord Configs below:

**CSVRecordReader** : Schema Access strategy as "Use 'Schema Text' Property

Schema Text: 


{
  "type": "record",
  "name": "AVLRecord0",
  "fields" : [
    {"name": "TimeOfDay","type": "string", "logicalType":"timestamp-millis"},
    {"name": "Field_0", "type": "double"},
    {"name": "Field_1", "type": "double"},
    {"name": "Field_2", "type": "double"},
    {"name": "Field_3", "type": "double"}}
]
}
**CSVRecordWritter**: 

Schema Write Strategy : Set 'Avro. schema' Attribute

Schema Access Strategy: Use Schema Text Property

Please tell me why am i not able to see the converted record after succesfully fetching from S3.

The desired output is CSV format only. Please find attached sample file uploaded on s3 and I want to convert only upto field_5.

enter image description here

Attached the contoller services screenshots:

enter image description here

enter image description here

enter image description here

Thank you!

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I have figured my error: 1. I forgot to add FetchS3Object Processor after EvaluateJsonPath 2. There was an extra comma in my Schema text Property.


  2. 0

    Can you tell where exactly was that extra comma in your convert record processor?
    As I am facing the same issue.
    As per my understanding, issue is occurring because of size_dimension field
    Below is my csv data :

    id,project,name,depth,parentid,description,createdtime,lastupdatedtime,metadata,path,source,sourceid
75125,abcd,P200184,4,74861,"WIRELINE RUNNING / RETRIEVING TOOL, SUPP",2002-06-04 00:00:00.0,2019-04-26 00:00:00.0,"{""material_group"":""group"",""weight_unit"":""LB"",""laboratory"":""PMC"",""object_type"":""material"",""pyspark_generated_time"":""2019-06-07, 13:32:20.287657"",""size_dimension"":""3'5""L X 3'5""W X 1'H"",""gross_weight"":""100.000"",""net_weight"":""100.000"",""valid_from_date"":""20031219""}","[59941,64249,74859,74861,75125]",RPA_SAA.MRA,P200184

    And the avro schema which i have used is:

    {
    "name":"abc",
    "namespace":"nifi",
    "type":"record",
    "fields": [
    {"name":"id", "type": ["long", "null"], "default": null},
    {"name":"project", "type": ["string", "null"], "default": null},
    {"name":"name", "type": ["string", "null"], "default": null},
    {"name":"depth", "type": ["int", "null"], "default": null},
    {"name":"parentid", "type": ["long", "null"], "default": null},
    {"name":"description", "type": ["string", "null"], "default": null},
    {"name":"createdtime","type": ["null",{ "type":"long", "logicalType":"timestamp-millis"}], "default":null},
    {"name":"lastupdatedtime","type": ["null",{ "type":"long", "logicalType":"timestamp-millis"}], "default":null},
    {"name":"metadata","type": ["string", "null"], "default": null},
    {"name":"path","type": ["string", "null"], "default": null},
    {"name":"source", "type": ["string", "null"], "default": null},
    {"name":"sourceid", "type": ["string", "null"], "default": null}
    ]
}
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search