I am testing out Google Cloud Run as a platform to run a new project. Project is developed using NodeJS and Laravel. Created a docker images based on php-fpm. This image runs fine on my dev environment running Ubuntu 21.04 and Docker 20.10.8.
When running the same image deployed on Google Cloud Run the application hangs randomly.
I have narrowed it down to a specific line in imported symfony http send function, class called from laravel index.php. Application will hang a function call to fast_cgi_finish_request()
. It will hang on that line 70% of cases and call to timeout after 300 seconds when nginx times out.
Hanging line in ./vendor/symfony/http-foundation/Response.php
:
public function send() {
$this->sendHeaders();
$this->sendContent();
if (function_exists('fastcgi_finish_request')) {
/* This line hangs 70% of the time. */
fastcgi_finish_request();
} elseif (!in_array(PHP_SAPI, ['cli', 'phpdbg'], true)) {
static::closeOutputBuffers(0, true);
}
return $this;
}
Since the same image works on other environments this is specific to Google Cloud Run enviornment. My current guess is some runtime resource since application can respond fine 1-10 times before haning application start to hang.
Any thoughts on how to debug this running in Google Cloud Run platform? I am feeling fairly limited in my debugging options at the moment. Let me know if more details are needed from setup.
Environment passed in to Laravel:
APP_LOG: errorlog
APP_URL: ...
APP_ENV: testing
APP_KEY: ....
APP_DEBUG: 'true'
LOG_CHANNEL: 'stderr'
CACHE_DRIVER: 'database'
SESSION_DRIVER: 'database'
DB...
Dockerfile:
FROM php:7.4-fpm as dev
# https://nginx.org/keys/nginx_signing.key
RUN apt-get update && apt-get install --no-install-recommends -y curl gnupg2 ca-certificates lsb-release && lsb_release && echo "deb http://nginx.org/packages/mainline/debian $(lsb_release -cs) nginx" > /etc/apt/sources.list.d/nginx.list &&
curl -o /tmp/nginx_signing.key https://nginx.org/keys/nginx_signing.key &&
mv /tmp/nginx_signing.key /etc/apt/trusted.gpg.d/nginx_signing.asc &&
apt-get update && apt-get install --no-install-recommends -y libpq-dev &&
docker-php-ext-install -j$(nproc) pdo_mysql pdo_pgsql &&
apt-get update && apt-get install --no-install-recommends -y nginx
# ----------------------
# Composer install step
# ----------------------
FROM composer:1.10 as build
WORKDIR /app
COPY ./composer.* ./artisan ./
COPY ./database ./database
COPY ./nova-components ./nova-components
COPY ./nova ./nova
## autoload resources
COPY ./bootstrap ./bootstrap
COPY ./app ./app
COPY ./routes ./routes
COPY ./config ./config
RUN composer install
--no-dev
--no-progress
--no-suggest
--no-interaction
--optimize-autoloader
--prefer-dist &&
composer dump-autoload
# ----------------------
# npm install step
# ----------------------
FROM node:14-alpine as node
WORKDIR /app
COPY ./*.json ./*.mix.js ./artisan /app/
COPY ./resources ./resources
COPY ./public ./public
RUN npm install &&
npm run build
# ----------------------
# The FPM production container
# ----------------------
FROM dev
WORKDIR /app
COPY ./docker/www.conf /usr/local/etc/php-fpm.d/www.conf
COPY ./docker/nginx.conf /etc/nginx/nginx.conf
COPY ./docker/entrypoint.sh /
COPY ./docker/php.ini-development ./docker/php.ini-production $PHP_INI_DIR/
COPY ./ /app
COPY --from=build /app/vendor /app/vendor
COPY --from=node /app/public/js/ /app/public/js/
COPY --from=node /app/public/mix-manifest.json /app/public/mix-manifest.json
RUN chmod +x /entrypoint.sh &&
rm -f /app/storage/logs/* /app/public/storage &&
php /app/artisan storage:link &&
mkdir /var/run/nginx &&
chmod -R 777 /app/storage /app/app /app/public/app &&
chown -R www-data:www-data /app/storage /app/app /app/public/app &&
chown -R www-data:www-data /var/log/nginx /var/cache/nginx /var/run/nginx
USER www-data
ENTRYPOINT ["/entrypoint.sh"]
composer show
:
64robots/nova-fields 0.18.0 A Laravel Nova field.
armincms/nova-tab 4.0.2 A Laravel Nova tool.
bernhardh/nova-translation-editor 1.3.1 Laravel Nova translation editor
fideloper/proxy 4.4.1 Set trusted proxies for Laravel
fruitcake/laravel-cors v2.0.4 Adds CORS (Cross-Origin Resource Sharing) headers support in your Laravel application
gkermer/nova-text-auto-complete 0.0.5 A Laravel Nova text autocomplete field.
guzzlehttp/guzzle 7.3.0 Guzzle is a PHP HTTP client library
intervention/image 2.6.0 Image handling and manipulation library with support for Laravel integration
laravel/framework v7.30.4 The Laravel Framework.
laravel/nova 3.16.3 A wonderful administration interface for Laravel.
laravel/tinker v2.6.1 Powerful REPL for the Laravel framework.
listen/nova-flexible-content dev-main Flexible Content & Repeater Fields for Laravel Nova.
listen/tree-view dev-main A Laravel Nova tool.
mcamara/laravel-localization 1.6.1 Easy localization for Laravel
ngiraud/nova-translatable-v2 1.0.4 A laravel-translatable extension for Laravel Nova. Inspired by optimistdigital/nova-translatable
optimistdigital/nova-settings 3.2.1 A Laravel Nova tool for editing custom settings using native Nova fields.
orangehill/iseed v3.0.1 Generate a new Laravel database seed file based on data from the existing database table.
silvanite/novatoolpermissions v1.1.3 Laravel Nova Permissions (Roles and Permission based Access Control (ACL))
waynestate/nova-ckeditor4-field 0.7.0 This nova package allows you to use CKEditor 4 for text areas.
yassi/nova-nested-form v3.0.12 A Laravel Nova package that allows you to create/update/delete nested related fields from a parent form.
nginx.conf:
pid /var/run/nginx/nginx.pid;
worker_processes auto;
events {
worker_connections 1024;
}
http {
include mime.types;
#include fastcgi.conf;
default_type application/octet-stream;
sendfile on;
tcp_nopush on;
server_tokens off;
client_max_body_size 10M;
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
server {
listen [::]:8080;
listen 8080 default_server;
server_name _;
root /app/public;
index index.php index.html index.htm;
access_log /dev/stdout;
error_log /dev/stdout info;
disable_symlinks off;
add_header X-Frame-Options "SAMEORIGIN";
add_header X-XSS-Protection "1; mode=block";
add_header X-Content-Type-Options "nosniff";
index index.html index.htm index.php;
charset utf-8;
include includes/*.conf;
location / {
try_files $uri $uri/ /index.php?$query_string;
}
location = /favicon.ico { access_log off; log_not_found off; }
location = /robots.txt { access_log off; log_not_found off; }
error_page 404 /index.php;
location ~ ^/status|^/ping {
fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
}
location ~ .php$ {
fastcgi_split_path_info ^(.+.php)(/.+)$;
fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
proxy_intercept_errors on;
fastcgi_intercept_errors on;
include fastcgi_params;
}
location ~ /.(?!well-known).* {
deny all;
}
}
}
Update
Something is definitly wrong with my setup and executing in cloud run. Enabled slowlog
and set request_slowlog_timeout
in php-fpm config. After timeout it releases to nginx and responds correctly. My app is realy slow with this setting but actually works. Missing something in fastcgi/php-fpm setup. Or perhaps Cloud Run is not a good fit for this project.
Update 2
After discussion in comments I migrated application to GKE standard where it is running without error since one week.
The problem persists even after the removal of secrets per @garbetjje suggestion. I have migrated to using the built in PHP web server. I have been running in Cloud Run for about 1.5 months now without issue. I realise that this is not a long term solution and am currently looking for other runtimes suitable for production use.
Update 3
Seems to be something related to first gen of Cloud Run. Deployed app in second generation Cloud Run service now. All functionality have not been thoroughly tested but seems app is working now..
2
Answers
I encountered the same problem recently (specifically, using Laravel Passport).
We were also making use of mounted secrets, and noticed that every 10-30 minutes our application would hang and time out after 60s (our configured timeout in Cloud Run). One of my colleagues noticed that every time our application started hanging, one of the first things to start failing was the reading of the mounted secret.
I asked about this behaviour in the Google Cloud Community Slack workspace, and one of the Cloud Run PM’s replied indicating that they were rolling back a recent implementation change.
I suspect that you encountered the same issue that we did (running our application in GKE also worked fine).
If it’s at all possible, switch to mounting your secrets into your environment. This should resolve the issues of your application hanging.
I have same problem. It’s weird that sometimes call small amount line of codes (get data from with eloquent model then send response), i get "upstream timeout".
Now Cloud Run has new different execution environment (beta) https://cloud.google.com/run/docs/about-execution-environments and i’m switching from initial first generation to second generation. Then my application behave normally.
I think the initial first generation have small issues:
Taken from docs: https://cloud.google.com/run/docs/about-execution-environments
Maybe this Github issue is correlated: https://github.com/google/gvisor/issues/158