I am trying to count the number of rows in an input file and I am using Cloud dataflow Runner for creating the template. In the below code, I am reading the file from a GCS bucket, processing it and…
I have a project that uses Google Dataflow. I have been successfully using the following command (and commands like it) for months to deploy templates. .venv/bin/python -m dataflow.registry_files.delimited_file --runner=DataflowRunner --region=us-central1 --project=myproject --staging_location=gs://mybucket-staging/staging/gr265 --template_location=gs://mybucket-code/templates/gr265 --temp_location=gs://mybucket-staging/temp/gr265 --no_use_public_ips --save_main_session --setup_file=dataflow/setup.py --projectId=myproject --datasetId=padl_staging --tableId=gr265…
My sample data is in json format and looks like: { "metadata": { "action": "insert", "type": "export", "version": 1, "timestamp": "2018-11-23T09:17:59.048-08:00" }, "data": { "attr1": 61, "day": "2018-11-22", "pin": "2C49956", "CDP": 0, "DP": 0, "VD": 0, "seo": 0, "dir": 0,…