Question 161858 in Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Why does JSONDecodeError occur when getting string from XCOM?

In a task, I serialise a dict (converting a dict to string) and I pushed it to XCOM

result[data] = json.dumps({"agents": ["[email protected]"], "houses": ["[email protected]"]})

In Airflow’s UI it looks good as a string, and the DAG level, I get in value
XCOM_DATA = "{{ task_instance.xcom_pull(task_ids='task_name', key='return_value')['data']}}"

But when a k8s pod operator, for some reason it deletes the double quotes(")

KubernetesPodOperator(
        cmds=["python", "src/send_notification.py"],
        arguments=[
            "--data",
            XCOM_DATA,
        ],
        task_id="notify",
        name="notify",
        dag=dag,
        **COMMON_TASKS_ARGS,
    )

UPDATE:
This is the command passed to K8S

 ['. /secrets/env; python /home/app/src/send_notification.py '
 '"--pr" '
 '"https://gitlab.com/30675" "--data" '
 '"{"agents": ["[email protected]", '
 '"[email protected]"]}" '

and when I deserialize it and inside the script, I get this error
json.loads(args.data)

obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

I printed the values and the error occurs for some reason airflow deletes the double quotes to the string and that’s why the error occurs. Can someone help me to correct it?

  • I used jinja methods like safe or escape
  • I convert to json in Jinja
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I fixed it adding | tojson

    XCOM_DATA = " "{{ task_instance.xcom_pull(task_ids='task_name', key='return_value')['data']| tojson " }} "

  2. 0

    You are sending string to the cli. that inside this string you need to add double quotes because of json format.

    in that case you need to send it with escape literals

    "{"agents": ["[email protected]","[email protected]"]}"
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search