Question posted in 
I have a flask application served by gunicorn.
All of this run inside a docker container.
When sending a POST request from the host (ubuntu) machine, on my computer, the server response is correct
On my VPS (debian 10), though, with the same docker image, I have an error from gunicorn (cf title)
[2024-02-28 17:08:13 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:538)
[2024-02-28 17:08:14 +0000] [1] [ERROR] Worker (pid:538) was sent SIGKILL! Perhaps out of memory?
[2024-02-28 17:08:14 +0000] [546] [INFO] Booting worker with pid: 546
Can you help me, I don’t know what to check… (maybe at docker level, but I don’t know how)
NB : I have a PostgresSQL database on the Host machine, and the access from within the container works fine (I can create the tables from inside the container with a flask db upgrade)
I tried to compare curl requests from MY ubuntu machine (correct response with a JSON indicating an invalid password)
curl -b "auth.strategy=local; auth.redirect=%2F; auth._token.local=false; auth._refresh_token.local=false" -d '{"email":"[email protected]","password":"12312312"}' -H "Content-Type: application/json" -X POST http://127.0.0.1:5000/icbf/api/v2/auth/login --verbose
* Trying 127.0.0.1:5000...
* Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)
> POST /icbf/api/v2/auth/login HTTP/1.1
> Host: 127.0.0.1:5000
> User-Agent: curl/7.81.0
> Accept: */*
> Cookie: auth.strategy=local; auth.redirect=%2F; auth._token.local=false; auth._refresh_token.local=false
> Content-Type: application/json
> Content-Length: 47
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 UNAUTHORIZED
< Server: gunicorn
< Date: Wed, 28 Feb 2024 17:07:28 GMT
< Connection: close
< Content-Type: application/json
< Content-Length: 47
< Access-Control-Allow-Origin: *
<
{"message": "Email ou mot de passe invalide."}
* Closing connection 0
and the one on the virtual server (VPS with debian 10), which is hanging for 20s before the error
curl -b "auth.strategy=local; auth.redirect=%2F; auth._token.local=false; auth._refresh_token.local=false" -d '{"email":"[email protected]","password":"12312312"}' -H "Content-Type: application/json" -X POST http://127.0.0.1:5000/icbf/api/v2/auth/login --verbose
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)
> POST /icbf/api/v2/auth/login HTTP/1.1
> Host: 127.0.0.1:5000
> User-Agent: curl/7.64.0
> Accept: */*
> Cookie: auth.strategy=local; auth.redirect=%2F; auth._token.local=false; auth._refresh_token.local=false
> Content-Type: application/json
> Content-Length: 47
>
* upload completely sent off: 47 out of 47 bytes
* Empty reply from server
* Connection #0 to host 127.0.0.1 left intact
curl: (52) Empty reply from server
I tried to check server memory with the free command from inside the VPS container, but it looks OK:
total used free shared buff/cache available
Mem: 1996412 441240 522812 34912 1032360 1357380
Swap: 0 0 0
I really don’t know where to look for…
EDIT : Strangely, gunicorn access log is empty…
Here is gunicorn complete configuration
gunicorn --print-config run:app
access_log_format = %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
accesslog = gunicorn_access_log
backlog = 2048
bind = ['0.0.0.0:5000']
ca_certs = None
capture_output = False
cert_reqs = 0
certfile = None
chdir = /api
check_config = False
child_exit = <ChildExit.child_exit()>
ciphers = None
config = ./gunicorn.conf.py
daemon = False
default_proc_name = run:app
disable_redirect_access_to_syslog = False
do_handshake_on_connect = False
dogstatsd_tags =
enable_stdio_inheritance = False
errorlog = gunicorn_err_log
forwarded_allow_ips = ['*']
graceful_timeout = 30
group = 0
initgroups = False
keepalive = 2
keyfile = None
limit_request_field_size = 8190
limit_request_fields = 100
limit_request_line = 4094
logconfig = None
logconfig_dict = {}
logconfig_json = None
logger_class = gunicorn.glogging.Logger
loglevel = debug
max_requests = 0
max_requests_jitter = 0
nworkers_changed = <NumWorkersChanged.nworkers_changed()>
on_exit = <OnExit.on_exit()>
on_reload = <OnReload.on_reload()>
on_starting = <OnStarting.on_starting()>
paste = None
pidfile = None
post_fork = <Postfork.post_fork()>
post_request = <PostRequest.post_request()>
post_worker_init = <PostWorkerInit.post_worker_init()>
pre_exec = <PreExec.pre_exec()>
pre_fork = <Prefork.pre_fork()>
pre_request = <PreRequest.pre_request()>
preload_app = False
print_config = True
proc_name = None
proxy_allow_ips = ['*']
proxy_protocol = False
pythonpath = None
raw_env = []
raw_paste_global_conf = []
reload = False
reload_engine = auto
reload_extra_files = []
reuse_port = False
secure_scheme_headers = {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
sendfile = None
spew = False
ssl_context = <NewSSLContext.ssl_context()>
ssl_version = 2
statsd_host = None
statsd_prefix =
strip_header_spaces = False
suppress_ragged_eofs = True
syslog = False
syslog_addr = udp://localhost:514
syslog_facility = user
syslog_prefix = None
threads = 1
timeout = 30
tmp_upload_dir = None
umask = 0
user = 0
when_ready = <WhenReady.when_ready()>
worker_abort = <WorkerAbort.worker_abort()>
worker_class = sync
worker_connections = 1000
worker_exit = <WorkerExit.worker_exit()>
worker_int = <WorkerInt.worker_int()>
worker_tmp_dir = /dev/shm
workers = 1
wsgi_app = None
2
Answers
I found the problem. 10 hours of tests just to realize the firewall is blocking part of the traffic to the db. with a
sudo ufw allow from 172.16.0.0/12 to any port 5432all containers have access to postgres and it 's all fine. Is there security issue with allowing all private ip range ??I have come across a similar situation. In my case, the problem was that at the time the workers started, they consumed more CPU power than was allowed for the virtual machine.
There were possible 2 ways to solve the problem:
Reducing the number of threads per worker, the number of requests per worker, or the number of workers themselves. (probably fewer workers will do just fine)
gunicorn -w <lesser_workers> –threads <lesser_threads>
Increasing the number of CPU cores for VM.