Why Gunicorn is serving Flask app fine on its own but throws error when invoked by systemd? – Nginx

Thank you in advance for any pointers.

I’ve got a Flask app wired with Gunicorn and Nginx. Gunicorn is serving the app OK after
running gunicorn --bind 0.0.0.0:5000 wsgi:app and I can see the app loaded in the browser.

But when invoked by systemd using project.service I get:

May 02 10:46:20 project gunicorn[1307]: Traceback (most recent call last):
May 02 10:46:20 project gunicorn[1307]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in s>
May 02 10:46:20 project gunicorn[1307]:     worker.init_process()
May 02 10:46:20 project gunicorn[1307]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134,>
May 02 10:46:20 project gunicorn[1307]:     self.load_wsgi()
May 02 10:46:20 project gunicorn[1307]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146,>
May 02 10:46:20 project gunicorn[1307]:     self.wsgi = self.app.wsgi()
May 02 10:46:20 project gunicorn[1307]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in w>
May 02 10:46:20 project gunicorn[1307]:     self.callable = self.load()
May 02 10:46:20 project gunicorn[1307]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, i>
May 02 10:46:20 project gunicorn[1307]:     return self.load_wsgiapp()
May 02 10:46:20 project gunicorn[1307]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, i>
May 02 10:46:20 project gunicorn[1307]:     return util.import_app(self.app_uri)
May 02 10:46:20 project gunicorn[1307]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/util.py", line 359, in impo>
May 02 10:46:20 project gunicorn[1307]:     mod = importlib.import_module(module)
May 02 10:46:20 project gunicorn[1307]:   File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
May 02 10:46:20 project gunicorn[1307]:     return _bootstrap._gcd_import(name[level:], package, level)
May 02 10:46:20 project gunicorn[1307]:   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
May 02 10:46:20 project gunicorn[1307]:   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
May 02 10:46:20 project gunicorn[1307]:   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
May 02 10:46:20 project gunicorn[1307]:   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
May 02 10:46:20 project gunicorn[1307]:   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
May 02 10:46:20 project gunicorn[1307]:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
May 02 10:46:20 project gunicorn[1307]:   File "/home/mark/project/wsgi.py", line 1, in <module>
May 02 10:46:20 project gunicorn[1307]:     from project import app
May 02 10:46:20 project gunicorn[1307]:   File "/home/mark/project/project/__init__.py", line 107, in <module>
May 02 10:46:20 project gunicorn[1307]:     app = create_app()
May 02 10:46:20 project gunicorn[1307]:   File "/home/mark/project/project/__init__.py", line 82, in create_app
May 02 10:46:20 project gunicorn[1307]:     db.init_app(app)
May 02 10:46:20 project gunicorn[1307]: UnboundLocalError: local variable 'app' referenced before assignment
May 02 10:46:20 project gunicorn[1307]: [2021-05-02 10:46:20 +0000] [1307] [INFO] Worker exiting (pid: 1307)
May 02 10:46:20 project gunicorn[1294]: Traceback (most recent call last):
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 209, in r>
May 02 10:46:20 project gunicorn[1294]:     self.sleep()
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 357, in s>
May 02 10:46:20 project gunicorn[1294]:     ready = select.select([self.PIPE[0]], [], [], 1.0)
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in h>
May 02 10:46:20 project gunicorn[1294]:     self.reap_workers()
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in r>
May 02 10:46:20 project gunicorn[1294]:     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
May 02 10:46:20 project gunicorn[1294]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
May 02 10:46:20 project gunicorn[1294]: During handling of the above exception, another exception occurred:
May 02 10:46:20 project gunicorn[1294]: Traceback (most recent call last):
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/bin/gunicorn", line 8, in <module>
May 02 10:46:20 project gunicorn[1294]:     sys.exit(run())
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, i>
May 02 10:46:20 project gunicorn[1294]:     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in >
May 02 10:46:20 project gunicorn[1294]:     super().run()
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in r>
May 02 10:46:20 project gunicorn[1294]:     Arbiter(self).run()
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 229, in r>
May 02 10:46:20 project gunicorn[1294]:     self.halt(reason=inst.reason, exit_status=inst.exit_status)
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 342, in h>
May 02 10:46:20 project gunicorn[1294]:     self.stop()
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 393, in s>
May 02 10:46:20 project gunicorn[1294]:     time.sleep(0.1)
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in h>
May 02 10:46:20 project gunicorn[1294]:     self.reap_workers()
May 02 10:46:20 project gunicorn[1294]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in r>
May 02 10:46:20 project gunicorn[1294]:     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
May 02 10:46:20 project gunicorn[1294]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
May 02 10:46:20 project systemd[1]: project.service: Main process exited, code=exited, status=1/FAILURE
May 02 10:46:20 project systemd[1]: project.service: Failed with result 'exit-code'.
May 02 10:46:28 project sudo[1310]:     mark : TTY=pts/0 ; PWD=/home/mark/project ; USER=root ; COMMAND=/usr/bin/systemctl daemon-reload
May 02 10:46:28 project sudo[1310]: pam_unix(sudo:session): session opened for user root by mark(uid=0)
May 02 10:46:28 project systemd[1]: Reloading.
May 02 10:46:28 project sudo[1310]: pam_unix(sudo:session): session closed for user root
May 02 10:46:36 project sudo[1342]:     mark : TTY=pts/0 ; PWD=/home/mark/project ; USER=root ; COMMAND=/usr/bin/systemctl status offenderb>
May 02 10:46:36 project sudo[1342]: pam_unix(sudo:session): session opened for user root by mark(uid=0)
May 02 10:46:42 project sshd[1345]: Invalid user pi from 81.165.210.66 port 45190
May 02 10:46:42 project sshd[1346]: Invalid user pi from 81.165.210.66 port 45194
May 02 10:46:42 project sshd[1345]: Connection closed by invalid user pi 81.165.210.66 port 45190 [preauth]
May 02 10:46:42 project sshd[1346]: Connection closed by invalid user pi 81.165.210.66 port 45194 [preauth]
May 02 10:46:46 project sudo[1342]: pam_unix(sudo:session): session closed for user root

I’ve observed this line UnboundLocalError: local variable 'app' referenced before assignment which isn’t showing up when Gunicorn runs on its own.

wsgi.py:

from project import app

if __name__ == '__main__':
    app.run(host='0.0.0.0')

project/init.py:

db = SQLAlchemy()


def create_app(script_info=None):

    if os.environ.get('FLASK_ENV') == 'production':
        app = Flask(__name__, ...)
        app.config.from_object(current_config)
        
    elif os.environ.get('FLASK_ENV') == 'staging':
        app = Flask(__name__, ...)
        app.config.from_object(current_config)
        
    elif os.environ.get('FLASK_ENV') == 'development':
        app = Flask(__name__, ...)
        app.config.from_object(current_config)
        toolbar.init_app(app)
        

    db.init_app(app)
    bcrypt.init_app(app)
    login_mgr.init_app(app)
    login_mgr.session_protection = "strong"
    ...
    ...  

    return app


app = create_app()

There’s env variable set export FLASK_ENV=staging

If you need more details please ask.

using Python 3.8.5 system-wide.

Edit:

after adding pi user error output is now:

May 02 18:33:08 project gunicorn[2029]:     from project import app
May 02 18:33:08 project gunicorn[2029]:   File "/home/mark/project/project/__init__.py", line 107, in <module>
May 02 18:33:08 project gunicorn[2029]:     app = create_app()
May 02 18:33:08 project gunicorn[2029]:   File "/home/mark/project/project/__init__.py", line 82, in create_app
May 02 18:33:08 project gunicorn[2029]:     db.init_app(app)
May 02 18:33:08 project gunicorn[2029]: UnboundLocalError: local variable 'app' referenced before assignment
May 02 18:33:08 project gunicorn[2029]: [2021-05-02 18:33:08 +0000] [2029] [INFO] Worker exiting (pid: 2029)
May 02 18:33:08 project gunicorn[2017]: Traceback (most recent call last):
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 209, in r>
May 02 18:33:08 project gunicorn[2017]:     self.sleep()
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 357, in s>
May 02 18:33:08 project gunicorn[2017]:     ready = select.select([self.PIPE[0]], [], [], 1.0)
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in h>
May 02 18:33:08 project gunicorn[2017]:     self.reap_workers()
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in r>
May 02 18:33:08 project gunicorn[2017]:     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
May 02 18:33:08 project gunicorn[2017]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
May 02 18:33:08 project gunicorn[2017]: During handling of the above exception, another exception occurred:
May 02 18:33:08 project gunicorn[2017]: Traceback (most recent call last):
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/bin/gunicorn", line 8, in <module>
May 02 18:33:08 project gunicorn[2017]:     sys.exit(run())
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, i>
May 02 18:33:08 project gunicorn[2017]:     WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in >
May 02 18:33:08 project gunicorn[2017]:     super().run()
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in r>
May 02 18:33:08 project gunicorn[2017]:     Arbiter(self).run()
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 229, in r>
May 02 18:33:08 project gunicorn[2017]:     self.halt(reason=inst.reason, exit_status=inst.exit_status)
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 342, in h>
May 02 18:33:08 project gunicorn[2017]:     self.stop()
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 393, in s>
May 02 18:33:08 project gunicorn[2017]:     time.sleep(0.1)
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 242, in h>
May 02 18:33:08 project gunicorn[2017]:     self.reap_workers()
May 02 18:33:08 project gunicorn[2017]:   File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 525, in r>
May 02 18:33:08 project gunicorn[2017]:     raise HaltServer(reason, self.WORKER_BOOT_ERROR)
May 02 18:33:08 project gunicorn[2017]: gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
May 02 18:33:08 project systemd[1]: project.service: Main process exited, code=exited, status=1/FAILURE
May 02 18:33:08 project systemd[1]: project.service: Failed with result 'exit-code'.
May 02 18:33:23 project sudo[2032]:     mark : TTY=pts/0 ; PWD=/home/mark ; USER=root ; COMMAND=/usr/bin/systemctl status project
May 02 18:33:23 project sudo[2032]: pam_unix(sudo:session): session opened for user root by mark(uid=0)
May 02 18:33:29 project sudo[2032]: pam_unix(sudo:session): session closed for user root
May 02 18:33:42 project sudo[2035]:     mark : TTY=pts/0 ; PWD=/home/mark ; USER=root ; COMMAND=/usr/bin/systemctl daemon-reload
May 02 18:33:42 project sudo[2035]: pam_unix(sudo:session): session opened for user root by mark(uid=0)
May 02 18:33:42 project systemd[1]: Reloading.
May 02 18:33:42 project sudo[2035]: pam_unix(sudo:session): session closed for user root
mark@project:~$

Another Edit:

I’ve added the pi user (even though I’m not on RPi) and added app = None. I also used Python debugger to get a better picture of what’s
happening. Python debugger revealed what I expected. Regular Flask app object is passed into db.init_app(app). Nothing suspicious.

Yet when I start off gunicorn by running gunicorn --preload --bind 0.0.0.0:5000 wsgi:app I get:

Traceback (most recent call last):
  File "/home/mark/project/staging/bin/gunicorn", line 8, in <module>
    sys.exit(run())
  File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
    super().run()
  File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 58, in __init__
    self.setup(app)
  File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/arbiter.py", line 118, in setup
    self.app.wsgi()
  File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/home/mark/project/staging/lib/python3.8/site-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/mark/project/wsgi.py", line 1, in <module>
    from project import app
ImportError: cannot import name 'app' from 'project' (/home/mark/project/project/__init__.py)

my wsgi.py file now looks like this:

from project import app

if __name__ == '__main__':
    app.run()

and __init__.py:

db = SQLAlchemy()


def create_app(script_info=None):
    app = None  # added
    if os.environ.get('FLASK_ENV') == 'production':
        app = Flask(__name__, ...)
        app.config.from_object(current_config)

    elif os.environ.get('FLASK_ENV') == 'staging':
        app = Flask(__name__, ...)
        app.config.from_object(current_config)

    elif os.environ.get('FLASK_ENV') == 'development':
        app = Flask(__name__, ...)
        app.config.from_object(current_config)
        toolbar.init_app(app)


    db.init_app(app)
    bcrypt.init_app(app)
    login_mgr.init_app(app)
    login_mgr.session_protection = "strong"
    ...
    ...  

    return app


if __name__ == '__main__':
    app = create_app()
    app.run(host='0.0.0.0')

running flask shell takes me into shell with Flask app context.
the below is as expected which is telling that the app object is being
returned OK:

Python 3.8.5 (default, Jan 27 2021, 15:41:15) 
[GCC 9.3.0] on linux
App: project [staging]
Instance: /home/mark/project/instance
>>> app
<Flask 'project'>
>>> dir(app)
['__call__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_before_request_lock', '_blueprint_order', '_find_error_handler', '_get_exc_class_and_code', '_got_first_request', '_register_error_handler', '_static_folder', '_static_url_path', 'add_template_filter', 'add_template_global', 'add_template_test', 'add_url_rule', 'after_request', 'after_request_funcs', 'app_context', 'app_ctx_globals_class', 'auto_find_instance_path', 'before_first_request', 'before_first_request_funcs', 'before_request', 'before_request_funcs', 'blueprints', 'cli', 'config', 'config_class', 'context_processor', 'create_global_jinja_loader', 'create_jinja_environment', 'create_url_adapter', 'debug', 'default_config', 'dispatch_request', 'do_teardown_appcontext', 'do_teardown_request', 'endpoint', 'env', 'error_handler_spec', 'errorhandler', 'extensions', 'finalize_request', 'full_dispatch_request', 'get_send_file_max_age', 'got_first_request', 'handle_exception', 'handle_http_exception', 'handle_url_build_error', 'handle_user_exception', 'has_static_folder', 'import_name', 'inject_url_defaults', 'instance_path', 'iter_blueprints', 'jinja_env', 'jinja_environment', 'jinja_loader', 'jinja_options', 'json_decoder', 'json_encoder', 'log_exception', 'logger', 'login_manager', 'make_config', 'make_default_options_response', 'make_null_session', 'make_response', 'make_shell_context', 'name', 'open_instance_resource', 'open_resource', 'open_session', 'permanent_session_lifetime', 'preprocess_request', 'preserve_context_on_exception', 'process_response', 'propagate_exceptions', 'raise_routing_exception', 'register_blueprint', 'register_error_handler', 'request_class', 'request_context', 'response_class', 'root_path', 'route', 'run', 'save_session', 'secret_key', 'select_jinja_autoescape', 'send_file_max_age_default', 'send_static_file', 'session_cookie_name', 'session_interface', 'shell_context_processor', 'shell_context_processors', 'should_ignore_error', 'static_folder', 'static_url_path', 'subdomain_matching', 'teardown_appcontext', 'teardown_appcontext_funcs', 'teardown_request', 'teardown_request_funcs', 'template_context_processors', 'template_filter', 'template_folder', 'template_global', 'template_test', 'templates_auto_reload', 'test_cli_runner', 'test_cli_runner_class', 'test_client', 'test_client_class', 'test_request_context', 'testing', 'trap_http_exception', 'try_trigger_before_first_request_functions', 'update_template_context', 'url_build_error_handlers', 'url_default_functions', 'url_defaults', 'url_map', 'url_map_class', 'url_rule_class', 'url_value_preprocessor', 'url_value_preprocessors', 'use_x_sendfile', 'view_functions', 'wsgi_app']