skip to Main Content

I have a project that uses Google Dataflow. I have been successfully using the following command (and commands like it) for months to deploy templates.

.venv/bin/python -m dataflow.registry_files.delimited_file --runner=DataflowRunner --region=us-central1 --project=myproject --staging_location=gs://mybucket-staging/staging/gr265 --template_location=gs://mybucket-code/templates/gr265 --temp_location=gs://mybucket-staging/temp/gr265 --no_use_public_ips --save_main_session --setup_file=dataflow/setup.py --projectId=myproject --datasetId=padl_staging --tableId=gr265 --configFile=gs://mybucket-code/registry/registry_files.yaml --fileType=gr265

This command continues to work on windows 10 and Debian machines in my team.

Since I upgraded to catalina (10.15.1, with python 3.7.5 and apache-beam==2.16.0 ) I get the following error:

[libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already exists in database: 
[libprotobuf FATAL google/protobuf/descriptor.cc:1370] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 
Abort trap: 6

I have done all of the following, with many reboots:

  1. Run xcode-select --install

  2. Run brew update-reset, brew update, brew upgrade, and brew reinstall python all to no effect (except after brew update-reset, brew doctor works again)

  3. Run brew uninstall protobuf and brew install protobuf

  4. Run pip3 uninstall protobuf outside of the virtual environments

  5. Deleted and re-created my virtual environments from their requirements files.

  6. Several bits of voodoo involving /usr/local/include that I located elsewhere on stack overflow that didn’t help.

I wondered if this was just my machine, and unfortunately was able to reproduce it on the other macOS Catalina laptop in my team, but not the laptop still running macOS Mojave.

2

Answers


  1. According with the Apache Beam Issue 8368 this issue is related with a pyarrow version, you have to test with pyarrow 0.15.1 beam since is the one which works on MacOS 10.15 as is mentioned in this link.

    Please let us know how it works.

    Login or Signup to reply.
  2. I just leave it here, because it is a common problem has not resolved by maintainers and I didn’t found the closest convenient solution in the time.

    https://github.com/protocolbuffers/protobuf/issues/1941

    https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=721791;filename=protobuf-2.4.1-3.1.debdiff;msg=5

    The patch actually quite does not help, because in the next step the protobuf does hang in the Run function and call does not return.

    The closest solution for myself was to completely avoid of any double linkage with the protobuf at any cost.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search