Ubuntu - Cannot import protobuf builder when deploying Azure function

JamesNaish
July 2, 2023
416 views
0 votes
2 Answers

The Problem

I have an Azure function built using the Python v1 programming model which builds a basic Keras model and trains it using some data from an Azure file share.

Tested locally and everything works.

I’m deploying to Azure using Azure Pipelines. It’s invoking correctly, but then failing. Checking the detailed invocation history reveals the following error:

Result: Failure Exception: ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/azure-functions-host/workers/python/3.9/LINUX/X64/google/protobuf/internal/__init__.py). 
Please check the requirements.txt file for the missing module. For more info, please refer the troubleshooting guide: https://aka.ms/functions-modulenotfound
Stack: File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 380, in _handle__function_load_request func = loader.load_function( 
File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 48, in call raise extend_exception_message(e, message) File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call return func(*args, **kwargs)
File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 132, in load_function mod = importlib.import_module(fullmodname)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked 
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed 
File "/home/site/wwwroot/func-gc-imgorientation-train/__init__.py", line 20, in <module> from core.data.training_image_filestore import TrainingImageFilestore
File "/home/site/wwwroot/core/data/training_image_filestore.py", line 4, in <module> import tensorflow as tf
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/__init__.py", line 37, in <module> from tensorflow.python.tools import module_util as _module_util
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/python/__init__.py", line 37, in <module> from tensorflow.python.eager import context
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/python/eager/context.py", line 28, in <module> from tensorflow.core.framework import function_pb2
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/core/framework/function_pb2.py", line 5, in <module> from google.protobuf.internal import builder as _builder

Well Documented Error with a Solution that Works Outside of Azure

This error is well documented and understood as in the following version:

ImportError: cannot import name 'builder' from 'google.protobuf.internal'

In general, arising from API changes in v3.20:

https://stackoverflow.com/a/71984564/1928761

TF adopted these changes, but there were bugs when using TF with protobuf >= v3.20 which were resolved in the most recent Tensorflow release so protobuf 4.23.3 should work with TF 2.12.0 as here:

https://github.com/tensorflow/tensorflow/issues/59221

Problem is resolved locally

I resolved this issue locally and have tested my function. All works fine. The problem only occurs in Azure.

Suspected Root Cause in Azure

Looking through the error above I noticed that all my custom modules, and Tensorflow, are installed to the same root: /home/site/wwwroot/

For instance, my custom core.data module is installed at /home/site/wwwroot/core/data/

Tensorflow is installed at /home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/

I’ve confirmed that all modules in my requirements.txt are being installed as expected in /home/site/wwwroot/.python_packages/lib/site-packages/ – including the latest protobuf with the builder module as expected.

However, protobuf is being imported from /azure-functions-host/workers/python/3.9/LINUX/X64/google/protobuf/internal/init.py

Presumably the implication is that TF is using the version of protobuf that’s bundled with Azure’s python distribution on the host, rather than the version in my site packages.

To test this theory I added the following code to the top of my init.py for the function:

import sys
print(sys.path)
import google.protobuf
print(google.protobuf.__version__)
print(google.protobuf.__path__)

This confirmed that protobuf version 3.19 was being loaded from the following path and not from site packages: /azure-functions-host/workers/python/3.9/LINUX/X64/google/protobuf/

The Question

The question therefore is can I upgrade the version of protobuf that’s included in this python bundle?

Alternatively, can I force TF to use the version of protobuf in my site packages rather than the ones in the Azure python bundle?

Attempted Solutions

ADO is building and installing the correct versions of both protobuf and TF, and the builder file is definitely in the installed site-packages. I’ve confirmed this by downloading the package from my storage account and unzipping it.

To resolve the issue, I’ve tried the following:

Confirmed that I can import other modules and that this is specific to importing the protobuf builder
Confirmed that I can import google.protobuf.internal.
Tried clearing the protobuf pycache by adding an rm -rvf command to my bash script immediately after the pip install –target….
Using subprocess to run pip install –upgrade protobuf==4.23.3 from within my function and before importing TF (which runs without error, but does not do the job).

At this point, I’m all out of ideas.

requirements.txt

# DO NOT include azure-functions-worker in this file
# The Python Worker is managed by Azure Functions platform
# Manually managing azure-functions-worker may cause unexpected issues

# Protobuf comes first to force the very latest version
protobuf==4.23.3

# Tensorflow, Dotenv, Pillow
absl-py==1.4.0
astunparse==1.6.3
cachetools==5.3.1
certifi==2023.5.7
charset-normalizer==3.1.0
contourpy==1.1.0
cycler==0.11.0
flatbuffers==23.5.26
fonttools==4.40.0
gast==0.4.0
google-auth==2.20.0
google-auth-oauthlib==1.0.0
google-pasta==0.2.0
graphviz==0.20.1
grpcio==1.54.2
h5py==3.8.0
idna==3.4
jax==0.4.12
keras==2.12.0
kiwisolver==1.4.4
libclang==16.0.0
Markdown==3.4.3
MarkupSafe==2.1.3
matplotlib==3.7.1
ml-dtypes==0.2.0
numpy==1.23.5
oauthlib==3.2.2
opt-einsum==3.3.0
packaging==23.1
Pillow==9.5.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pydot==1.4.2
pyparsing==3.0.9
python-dateutil==2.8.2
python-dotenv
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
scipy==1.10.1
six==1.16.0
tensorboard==2.12.3
tensorboard-data-server==0.7.1
tensorflow==2.12.0
tensorflow-estimator==2.12.0
termcolor==2.3.0
typing_extensions==4.6.3
urllib3==1.26.16
Werkzeug==2.3.6
wrapt==1.14.1

# Azure libraries
azure-functions
azure-identity
azure-keyvault

Deployment pipeline YAML

# Python Function App to Linux on Azure
# Build a Python function app and deploy it to Azure as a Linux function app.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/python

trigger:
- main

variables:
  # Azure Resource Manager connection created during pipeline creation
  azureSubscription: '4ae24131-0b22-421c-8e3e-6d766e891ece'

  # Function app name
  functionAppName: 'func-xxxx-dev'

  # Agent VM image name
  vmImageName: 'ubuntu-latest'

  # Working Directory
  workingDirectory: '$(System.DefaultWorkingDirectory)'

stages:
- stage: Build
  displayName: Build stage

  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)

    steps:
    - bash: |
        if [ -f extensions.csproj ]
        then
            dotnet build extensions.csproj --runtime ubuntu.16.04-x64 --output ./bin
        fi
      workingDirectory: $(workingDirectory)
      displayName: 'Build extensions'

    - task: UsePythonVersion@0
      displayName: 'Use Python 3.9'
      inputs:
        versionSpec: 3.9 # Functions V2 supports Python 3.6 as of today

    - bash: |
        python -m pip install --upgrade pip
        pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt
      workingDirectory: $(workingDirectory)
      displayName: 'Install application dependencies'

    - task: ArchiveFiles@2
      displayName: 'Archive files'
      inputs:
        rootFolderOrFile: '$(workingDirectory)'
        includeRootFolder: false
        archiveType: zip
        archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
        replaceExistingArchive: true

    - publish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
      artifact: drop

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build
  condition: succeeded()

  jobs:
  - deployment: Deploy
    displayName: Deploy
    environment: 'development'
    pool:
      vmImage: $(vmImageName)

    strategy:
      runOnce:
        deploy:

          steps:
          - task: AzureFunctionApp@1
            displayName: 'Azure functions app deploy'
            inputs:
              azureSubscription: '$(azureSubscription)'
              appType: functionAppLinux
              appName: $(functionAppName)
              package: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'

Log stream

2023-06-27T11:50:43Z   [Information]   Host Status: {
  "id": "func-xxxx-dev",
  "state": "Running",
  "version": "4.21.3.3",
  "versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
  "platformVersion": "",
  "instanceId": "9A4EF22A-638234606489642041",
  "computerName": "",
  "processUptime": 2778401,
  "functionAppContentEditingState": "NotAllowed",
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "4.5.0"
  }
}
2023-06-27T11:50:43Z   [Information]   Host Status: {
  "id": "func-xxxx-dev",
  "state": "Running",
  "version": "4.21.3.3",
  "versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
  "platformVersion": "",
  "instanceId": "9A4EF22A-638234606489642041",
  "computerName": "",
  "processUptime": 2778404,
  "functionAppContentEditingState": "NotAllowed",
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "4.5.0"
  }
}
2023-06-27T11:50:43Z   [Information]   Host Status: {
  "id": "func-xxxx-dev",
  "state": "Running",
  "version": "4.21.3.3",
  "versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
  "platformVersion": "",
  "instanceId": "9A4EF22A-638234606489642041",
  "computerName": "",
  "processUptime": 2778801,
  "functionAppContentEditingState": "NotAllowed",
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "4.5.0"
  }
}
2023-06-27T11:51:00Z   [Information]   Executing 'Functions.func-xxxx-train' (Reason='Timer fired at 2023-06-27T11:51:00.0016150+00:00', Id=3795600b-379e-423c-b29f-65fec390289a)
2023-06-27T11:51:00Z   [Verbose]   Sending invocation id: '3795600b-379e-423c-b29f-65fec390289a
2023-06-27T11:51:00Z   [Verbose]   Posting invocation id:3795600b-379e-423c-b29f-65fec390289a on workerId:250ae2e1-4416-41ef-b355-0684a59d0a91
2023-06-27T11:51:00Z   [Error]   Executed 'Functions.func-xxxx-train' (Failed, Id=3795600b-379e-423c-b29f-65fec390289a, Duration=2ms)
2023-06-27T11:51:00Z   [Verbose]   Function 'func-xxx-train' updated status: Last='2023-06-27T11:51:00.0015724+00:00', Next='2023-06-27T11:52:00.0000000+00:00', LastUpdated='2023-06-27T11:51:00.0015724+00:00'
2023-06-27T11:51:00Z   [Verbose]   Timer for 'func-gc-imgorientation-train' started with interval '00:00:59.9674482'.
2023-06-27T11:51:05Z   [Information]   Host Status: {
  "id": "func-gc-imgorientation-dev",
  "state": "Running",
  "version": "4.21.3.3",
  "versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
  "platformVersion": "",
  "instanceId": "9A4EF22A-638234606489642041",
  "computerName": "",
  "processUptime": 2800545,
  "functionAppContentEditingState": "NotAllowed",
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "4.5.0"
  }
}

Answers

Chosen as BEST ANSWER
- JamesNaish
- July 2, 2023 at 10:26 am
- 0 votes
0
The root cause of the problem, as highlighted by @SiddheshDesai, turned out to be that the Azure Functions host was loading a version of protobuf (3.19.6) into the cache which was earlier than the version that the latest Tensorflow needed (4.23.3).

Even when placing my protobuf/TF imports at the top of my function app module, the same error arose.

I was, however, able to reload protobuf from my site-packages using importlib.

This cause was proven by adding the following code to my function:
```
import logging
import importlib
import google.protobuf

logging.info(google.protobuf.__version__)
logging.info(google.protobuf.__file__)

importlib.reload(google.protobuf)

logging.info(google.protobuf.__version__)
logging.info(google.protobuf.__file__)
```
However, the reload then interfered with the Azure Functions library and host causing a heap of other exceptions.

My conclusion therefore is that this is an issue with Azure Functions rather than an issue with my code.

Rather than trying to work around these issues, I've now containerised my function and re-deployed. This has worked as expected.

The answer, therefore, turns out to be - as SiddheshDesai suggested above - that any application needing to use versions of protobuf >= 3.20.0 need to be containerised rather than deployed as code/run from packages.

(Edit)

If you want to use Protobuf module in Azure Functions, You need to downgrade it to 3.20.* and add it in your requirements.txt.

I added protobuf==3.20.* in my requirements.txt and the Http Trigger got deployed successfully in Azure Functions via DevOps YAML pipeline, Refer below:-

My requirements.txt:-

azure-functions
protobuf==3.20.*

My init.py:-

import logging

import azure.functions as func


def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    name = req.params.get('name')
    if not name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            name = req_body.get('name')

    if name:
        return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed successfully.")
    else:
        return func.HttpResponse(
             "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
             status_code=200
        )

My YAML pipeline:-

trigger:
- master

variables:
  
  azureSubscription: 'xxxxxxxx-xxxxx-xxx9bbd4354dd'

 
  functionAppName: 'valleyfunc541'

 
  vmImageName: 'ubuntu-latest'

 
  workingDirectory: '$(System.DefaultWorkingDirectory)'

stages:
- stage: Build
  displayName: Build stage

  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)

    steps:
    - bash: |
        if [ -f extensions.csproj ]
        then
            dotnet build extensions.csproj --runtime ubuntu.16.04-x64 --output ./bin
        fi
      workingDirectory: $(workingDirectory)
      displayName: 'Build extensions'

    - task: UsePythonVersion@0
      displayName: 'Use Python 3.10'
      inputs:
        versionSpec: 3.10 # Functions V2 supports Python 3.6 as of today

    - bash: |
        pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt
      workingDirectory: $(workingDirectory)
      displayName: 'Install application dependencies'

    - task: ArchiveFiles@2
      displayName: 'Archive files'
      inputs:
        rootFolderOrFile: '$(workingDirectory)'
        includeRootFolder: false
        archiveType: zip
        archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
        replaceExistingArchive: true

    - publish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
      artifact: drop

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build
  condition: succeeded()

  jobs:
  - deployment: Deploy
    displayName: Deploy
    environment: 'development'
    pool:
      vmImage: $(vmImageName)

    strategy:
      runOnce:
        deploy:

          steps:
          - task: AzureFunctionApp@1
            displayName: 'Azure functions app deploy'
            inputs:
              azureSubscription: '$(azureSubscription)'
              appType: functionAppLinux
              appName: $(functionAppName)
              package: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'

Ubuntu – Cannot import protobuf builder when deploying Azure function

Answers