The Problem
I have an Azure function built using the Python v1 programming model which builds a basic Keras model and trains it using some data from an Azure file share.
Tested locally and everything works.
I’m deploying to Azure using Azure Pipelines. It’s invoking correctly, but then failing. Checking the detailed invocation history reveals the following error:
Result: Failure Exception: ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/azure-functions-host/workers/python/3.9/LINUX/X64/google/protobuf/internal/__init__.py).
Please check the requirements.txt file for the missing module. For more info, please refer the troubleshooting guide: https://aka.ms/functions-modulenotfound
Stack: File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 380, in _handle__function_load_request func = loader.load_function(
File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 48, in call raise extend_exception_message(e, message) File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call return func(*args, **kwargs)
File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 132, in load_function mod = importlib.import_module(fullmodname)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/home/site/wwwroot/func-gc-imgorientation-train/__init__.py", line 20, in <module> from core.data.training_image_filestore import TrainingImageFilestore
File "/home/site/wwwroot/core/data/training_image_filestore.py", line 4, in <module> import tensorflow as tf
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/__init__.py", line 37, in <module> from tensorflow.python.tools import module_util as _module_util
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/python/__init__.py", line 37, in <module> from tensorflow.python.eager import context
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/python/eager/context.py", line 28, in <module> from tensorflow.core.framework import function_pb2
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/core/framework/function_pb2.py", line 5, in <module> from google.protobuf.internal import builder as _builder
Well Documented Error with a Solution that Works Outside of Azure
This error is well documented and understood as in the following version:
ImportError: cannot import name 'builder' from 'google.protobuf.internal'
In general, arising from API changes in v3.20:
https://stackoverflow.com/a/71984564/1928761
TF adopted these changes, but there were bugs when using TF with protobuf >= v3.20 which were resolved in the most recent Tensorflow release so protobuf 4.23.3 should work with TF 2.12.0 as here:
https://github.com/tensorflow/tensorflow/issues/59221
Problem is resolved locally
I resolved this issue locally and have tested my function. All works fine. The problem only occurs in Azure.
Suspected Root Cause in Azure
Looking through the error above I noticed that all my custom modules, and Tensorflow, are installed to the same root: /home/site/wwwroot/
For instance, my custom core.data module is installed at /home/site/wwwroot/core/data/
Tensorflow is installed at /home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/
I’ve confirmed that all modules in my requirements.txt are being installed as expected in /home/site/wwwroot/.python_packages/lib/site-packages/ – including the latest protobuf with the builder module as expected.
However, protobuf is being imported from /azure-functions-host/workers/python/3.9/LINUX/X64/google/protobuf/internal/init.py
Presumably the implication is that TF is using the version of protobuf that’s bundled with Azure’s python distribution on the host, rather than the version in my site packages.
To test this theory I added the following code to the top of my init.py for the function:
import sys
print(sys.path)
import google.protobuf
print(google.protobuf.__version__)
print(google.protobuf.__path__)
This confirmed that protobuf version 3.19 was being loaded from the following path and not from site packages: /azure-functions-host/workers/python/3.9/LINUX/X64/google/protobuf/
The Question
The question therefore is can I upgrade the version of protobuf that’s included in this python bundle?
Alternatively, can I force TF to use the version of protobuf in my site packages rather than the ones in the Azure python bundle?
Attempted Solutions
ADO is building and installing the correct versions of both protobuf and TF, and the builder file is definitely in the installed site-packages. I’ve confirmed this by downloading the package from my storage account and unzipping it.
To resolve the issue, I’ve tried the following:
- Confirmed that I can import other modules and that this is specific to importing the protobuf builder
- Confirmed that I can import google.protobuf.internal.
- Tried clearing the protobuf pycache by adding an rm -rvf command to my bash script immediately after the pip install –target….
- Using subprocess to run pip install –upgrade protobuf==4.23.3 from within my function and before importing TF (which runs without error, but does not do the job).
At this point, I’m all out of ideas.
requirements.txt
# DO NOT include azure-functions-worker in this file
# The Python Worker is managed by Azure Functions platform
# Manually managing azure-functions-worker may cause unexpected issues
# Protobuf comes first to force the very latest version
protobuf==4.23.3
# Tensorflow, Dotenv, Pillow
absl-py==1.4.0
astunparse==1.6.3
cachetools==5.3.1
certifi==2023.5.7
charset-normalizer==3.1.0
contourpy==1.1.0
cycler==0.11.0
flatbuffers==23.5.26
fonttools==4.40.0
gast==0.4.0
google-auth==2.20.0
google-auth-oauthlib==1.0.0
google-pasta==0.2.0
graphviz==0.20.1
grpcio==1.54.2
h5py==3.8.0
idna==3.4
jax==0.4.12
keras==2.12.0
kiwisolver==1.4.4
libclang==16.0.0
Markdown==3.4.3
MarkupSafe==2.1.3
matplotlib==3.7.1
ml-dtypes==0.2.0
numpy==1.23.5
oauthlib==3.2.2
opt-einsum==3.3.0
packaging==23.1
Pillow==9.5.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pydot==1.4.2
pyparsing==3.0.9
python-dateutil==2.8.2
python-dotenv
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
scipy==1.10.1
six==1.16.0
tensorboard==2.12.3
tensorboard-data-server==0.7.1
tensorflow==2.12.0
tensorflow-estimator==2.12.0
termcolor==2.3.0
typing_extensions==4.6.3
urllib3==1.26.16
Werkzeug==2.3.6
wrapt==1.14.1
# Azure libraries
azure-functions
azure-identity
azure-keyvault
Deployment pipeline YAML
# Python Function App to Linux on Azure
# Build a Python function app and deploy it to Azure as a Linux function app.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/python
trigger:
- main
variables:
# Azure Resource Manager connection created during pipeline creation
azureSubscription: '4ae24131-0b22-421c-8e3e-6d766e891ece'
# Function app name
functionAppName: 'func-xxxx-dev'
# Agent VM image name
vmImageName: 'ubuntu-latest'
# Working Directory
workingDirectory: '$(System.DefaultWorkingDirectory)'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- bash: |
if [ -f extensions.csproj ]
then
dotnet build extensions.csproj --runtime ubuntu.16.04-x64 --output ./bin
fi
workingDirectory: $(workingDirectory)
displayName: 'Build extensions'
- task: UsePythonVersion@0
displayName: 'Use Python 3.9'
inputs:
versionSpec: 3.9 # Functions V2 supports Python 3.6 as of today
- bash: |
python -m pip install --upgrade pip
pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt
workingDirectory: $(workingDirectory)
displayName: 'Install application dependencies'
- task: ArchiveFiles@2
displayName: 'Archive files'
inputs:
rootFolderOrFile: '$(workingDirectory)'
includeRootFolder: false
archiveType: zip
archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
replaceExistingArchive: true
- publish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
artifact: drop
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
condition: succeeded()
jobs:
- deployment: Deploy
displayName: Deploy
environment: 'development'
pool:
vmImage: $(vmImageName)
strategy:
runOnce:
deploy:
steps:
- task: AzureFunctionApp@1
displayName: 'Azure functions app deploy'
inputs:
azureSubscription: '$(azureSubscription)'
appType: functionAppLinux
appName: $(functionAppName)
package: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'
Log stream
2023-06-27T11:50:43Z [Information] Host Status: {
"id": "func-xxxx-dev",
"state": "Running",
"version": "4.21.3.3",
"versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
"platformVersion": "",
"instanceId": "9A4EF22A-638234606489642041",
"computerName": "",
"processUptime": 2778401,
"functionAppContentEditingState": "NotAllowed",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "4.5.0"
}
}
2023-06-27T11:50:43Z [Information] Host Status: {
"id": "func-xxxx-dev",
"state": "Running",
"version": "4.21.3.3",
"versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
"platformVersion": "",
"instanceId": "9A4EF22A-638234606489642041",
"computerName": "",
"processUptime": 2778404,
"functionAppContentEditingState": "NotAllowed",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "4.5.0"
}
}
2023-06-27T11:50:43Z [Information] Host Status: {
"id": "func-xxxx-dev",
"state": "Running",
"version": "4.21.3.3",
"versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
"platformVersion": "",
"instanceId": "9A4EF22A-638234606489642041",
"computerName": "",
"processUptime": 2778801,
"functionAppContentEditingState": "NotAllowed",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "4.5.0"
}
}
2023-06-27T11:51:00Z [Information] Executing 'Functions.func-xxxx-train' (Reason='Timer fired at 2023-06-27T11:51:00.0016150+00:00', Id=3795600b-379e-423c-b29f-65fec390289a)
2023-06-27T11:51:00Z [Verbose] Sending invocation id: '3795600b-379e-423c-b29f-65fec390289a
2023-06-27T11:51:00Z [Verbose] Posting invocation id:3795600b-379e-423c-b29f-65fec390289a on workerId:250ae2e1-4416-41ef-b355-0684a59d0a91
2023-06-27T11:51:00Z [Error] Executed 'Functions.func-xxxx-train' (Failed, Id=3795600b-379e-423c-b29f-65fec390289a, Duration=2ms)
2023-06-27T11:51:00Z [Verbose] Function 'func-xxx-train' updated status: Last='2023-06-27T11:51:00.0015724+00:00', Next='2023-06-27T11:52:00.0000000+00:00', LastUpdated='2023-06-27T11:51:00.0015724+00:00'
2023-06-27T11:51:00Z [Verbose] Timer for 'func-gc-imgorientation-train' started with interval '00:00:59.9674482'.
2023-06-27T11:51:05Z [Information] Host Status: {
"id": "func-gc-imgorientation-dev",
"state": "Running",
"version": "4.21.3.3",
"versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
"platformVersion": "",
"instanceId": "9A4EF22A-638234606489642041",
"computerName": "",
"processUptime": 2800545,
"functionAppContentEditingState": "NotAllowed",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "4.5.0"
}
}
2
Answers
The root cause of the problem, as highlighted by @SiddheshDesai, turned out to be that the Azure Functions host was loading a version of protobuf (3.19.6) into the cache which was earlier than the version that the latest Tensorflow needed (4.23.3).
Even when placing my protobuf/TF imports at the top of my function app module, the same error arose.
I was, however, able to reload protobuf from my site-packages using importlib.
This cause was proven by adding the following code to my function:
However, the reload then interfered with the Azure Functions library and host causing a heap of other exceptions.
My conclusion therefore is that this is an issue with Azure Functions rather than an issue with my code.
Rather than trying to work around these issues, I've now containerised my function and re-deployed. This has worked as expected.
The answer, therefore, turns out to be - as SiddheshDesai suggested above - that any application needing to use versions of protobuf >= 3.20.0 need to be containerised rather than deployed as code/run from packages.
If you want to use Protobuf module in Azure Functions, You need to downgrade it to 3.20.* and add it in your requirements.txt.
I added protobuf==3.20.* in my requirements.txt and the Http Trigger got deployed successfully in Azure Functions via DevOps YAML pipeline, Refer below:-
My requirements.txt:-
My init.py:-
My YAML pipeline:-
Output:-
The HTTP Trigger got deployed successfully:-
Reference:- My SO thread answer