Debian - Docker-Compose failing to Build Container when using postgresql

Martin
September 26, 2024
216 views
0 votes
2 Answers

I have been trying to build a docker-compose container for my back-end service using Fastapi.The issue is docker-compose container logs keep returning that my database engine fails. I am using asyncpg for async connection in postgresql.


 ╰─λ sudo docker-compose logs
db-1  | The files belonging to this database system will be owned by user "postgres".
db-1  | This user must also own the server process.
db-1  | 
db-1  | The database cluster will be initialized with locale "en_US.utf8".
db-1  | The default database encoding has accordingly been set to "UTF8".
db-1  | The default text search configuration will be set to "english".
db-1  | 
db-1  | Data page checksums are disabled.
db-1  | 
db-1  | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db-1  | creating subdirectories ... ok
db-1  | selecting dynamic shared memory implementation ... posix
db-1  | selecting default max_connections ... 100
db-1  | selecting default shared_buffers ... 128MB
db-1  | selecting default time zone ... Etc/UTC
db-1  | creating configuration files ... ok
db-1  | running bootstrap script ... ok
db-1  | performing post-bootstrap initialization ... ok
db-1  | syncing data to disk ... ok
db-1  | 
db-1  | 
db-1  | initdb: warning: enabling "trust" authentication for local connections
db-1  | You can change this by editing pg_hba.conf or using the option -A, or
db-1  | --auth-local and --auth-host, the next time you run initdb.
db-1  | Success. You can now start the database server using:
db-1  | 
db-1  |     pg_ctl -D /var/lib/postgresql/data -l logfile start
db-1  | 
db-1  | waiting for server to start....2024-09-26 07:07:33.922 UTC [48] LOG:  starting PostgreSQL 13.16 (Debian 13.16-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
db-1  | 2024-09-26 07:07:33.928 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db-1  | 2024-09-26 07:07:33.945 UTC [49] LOG:  database system was shut down at 2024-09-26 07:07:30 UTC
db-1  | 2024-09-26 07:07:33.964 UTC [48] LOG:  database system is ready to accept connections
db-1  |  done
db-1  | server started
db-1  | CREATE DATABASE
db-1  | 
db-1  | 
db-1  | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db-1  | 
db-1  | 2024-09-26 07:07:35.936 UTC [48] LOG:  received fast shutdown request
db-1  | waiting for server to shut down....2024-09-26 07:07:35.943 UTC [48] LOG:  aborting any active transactions
db-1  | 2024-09-26 07:07:35.954 UTC [48] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1
db-1  | 2024-09-26 07:07:35.961 UTC [50] LOG:  shutting down
db-1  | 2024-09-26 07:07:36.021 UTC [48] LOG:  database system is shut down
db-1  |  done
db-1  | server stopped
db-1  | 
db-1  | PostgreSQL init process complete; ready for start up.
db-1  | 
db-1  | 2024-09-26 07:07:36.133 UTC [1] LOG:  starting PostgreSQL 13.16 (Debian 13.16-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
app-1  | /usr/local/lib/python3.12/site-packages/pydantic/_internal/_config.py:341: UserWarning: Valid config keys have changed in V2:
app-1  | * 'orm_mode' has been renamed to 'from_attributes'
app-1  |   warnings.warn(message, UserWarning)
db-1   | 2024-09-26 07:07:36.134 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db-1   | 2024-09-26 07:07:36.134 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db-1   | 2024-09-26 07:07:36.146 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db-1   | 2024-09-26 07:07:36.166 UTC [63] LOG:  database system was shut down at 2024-09-26 07:07:35 UTC
db-1   | 2024-09-26 07:07:36.195 UTC [1] LOG:  database system is ready to accept connections
app-1  | 2024-09-26 07:07:32.494 | INFO     | app.database.engine:get_engine:11 - Creating new database engine
app-1  | Traceback (most recent call last):
app-1  |   File "/usr/local/bin/uvicorn", line 8, in <module>
app-1  |     sys.exit(main())
app-1  |              ^^^^^^
app-1  |   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
app-1  |     return self.main(*args, **kwargs)
app-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1078, in main
app-1  |     rv = self.invoke(ctx)
app-1  |          ^^^^^^^^^^^^^^^^
app-1  |   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
app-1  |     return ctx.invoke(self.callback, **ctx.params)
app-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 783, in invoke
app-1  |     return __callback(*args, **kwargs)
app-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/usr/local/lib/python3.12/site-packages/uvicorn/main.py", line 410, in main
app-1  |     run(
app-1  |   File "/usr/local/lib/python3.12/site-packages/uvicorn/main.py", line 577, in run
app-1  |     server.run()
app-1  |   File "/usr/local/lib/python3.12/site-packages/uvicorn/server.py", line 65, in run
app-1  |     return asyncio.run(self.serve(sockets=sockets))
app-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
app-1  |     return runner.run(main)
app-1  |            ^^^^^^^^^^^^^^^^
app-1  |   File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
app-1  |     return self._loop.run_until_complete(task)
app-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
app-1  |   File "/usr/local/lib/python3.12/site-packages/uvicorn/server.py", line 69, in serve
app-1  |     await self._serve(sockets)
app-1  |   File "/usr/local/lib/python3.12/site-packages/uvicorn/server.py", line 76, in _serve
app-1  |     config.load()
app-1  |   File "/usr/local/lib/python3.12/site-packages/uvicorn/config.py", line 434, in load
app-1  |     self.loaded_app = import_from_string(self.app)
app-1  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/usr/local/lib/python3.12/site-packages/uvicorn/importer.py", line 22, in import_from_string
app-1  |     raise exc from None
app-1  |   File "/usr/local/lib/python3.12/site-packages/uvicorn/importer.py", line 19, in import_from_string
app-1  |     module = importlib.import_module(module_str)
app-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
app-1  |     return _bootstrap._gcd_import(name[level:], package, level)
app-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
app-1  |   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
app-1  |   File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
app-1  |   File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
app-1  |   File "<frozen importlib._bootstrap_external>", line 995, in exec_module
app-1  |   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
app-1  |   File "/app/main.py", line 1, in <module>
app-1  |     from app.api.routes.router import base_router as router
app-1  |   File "/app/app/api/routes/router.py", line 2, in <module>
app-1  |     from . import users, signup, login, orders, sms
app-1  |   File "/app/app/api/routes/users.py", line 6, in <module>
app-1  |     from app.database.session import get_db_session
app-1  |   File "/app/app/database/session.py", line 14, in <module>
app-1  |     from app.database.engine import get_engine
app-1  |   File "/app/app/database/engine.py", line 27, in <module>
app-1  |     engine = EngineManager.get_engine()
app-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/app/app/database/engine.py", line 12, in get_engine
app-1  |     cls._engine = create_async_engine(
app-1  |                   ^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/usr/local/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/engine.py", line 120, in create_async_engine
app-1  |     sync_engine = _create_engine(url, **kw)
app-1  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "<string>", line 2, in create_engine
app-1  |   File "/usr/local/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
app-1  |     return fn(*args, **kwargs)  # type: ignore[no-any-return]
app-1  |            ^^^^^^^^^^^^^^^^^^^
app-1  |   File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/create.py", line 599, in create_engine
app-1  |     dbapi = dbapi_meth(**dbapi_args)
app-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/usr/local/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 690, in import_dbapi
app-1  |     import psycopg2
app-1  | ModuleNotFoundError: No module named 'psycopg2'

the code from my engine class and session manager is as follows

from sqlalchemy.ext.asyncio import create_async_engine, AsyncEngine
from app.config import settings
from loguru import logger

class EngineManager:
    _engine: AsyncEngine | None = None

    @classmethod
    def get_engine(cls) -> AsyncEngine:
        if cls._engine is None:
            logger.info("Creating new database engine")
            cls._engine = create_async_engine(
                settings.database_url,
                echo=True,  # Set to False in production
                future=True,
            )
        return cls._engine

    @classmethod
    async def close_engine(cls):
        if cls._engine:
            logger.info("Closing database engine")
            await cls._engine.dispose()
            cls._engine = None

# Create a global instance of the engine
engine = EngineManager.get_engine()

# Function to get the engine (can be used in other parts of the application)
def get_engine() -> AsyncEngine:
    return EngineManager.get_engine()

from sqlalchemy.ext.asyncio import (
    AsyncSession,
    AsyncEngine,
    AsyncConnection,
    create_async_engine,
    async_sessionmaker
)
from fastapi import HTTPException
import contextlib
from app.config import settings
from typing import AsyncIterator
from loguru import logger
from sqlalchemy.exc import SQLAlchemyError
from app.database.engine import get_engine

class DataSessionManager:
    # database session config
    def  __init__(self):
        self.engine: AsyncEngine = get_engine()
        self._sessionmaker:  async_sessionmaker[AsyncSession] = async_sessionmaker(autocommit=False, bind=self.engine)

    
    async def close(self):
        if self.engine is None:
            raise HTTPException(status_code=404, message="service unavailable!")
        await self.engine.dispose()
        self.engine = None
        self._sessionmaker = None

    @contextlib.asynccontextmanager
    async def connection(self) -> AsyncIterator[AsyncConnection]:
        if self.engine is None:
            raise HTTPException(status_code=404, message="Service not found!")
        
        async with self.engine.begin() as connection:
            try:
                yield connection
            except SQLAlchemyError:
                await connection.rollback()
                logger.error("Connection error occured")
                raise HTTPException(status_code=404)


    @contextlib.asynccontextmanager
    async def session(self) -> AsyncIterator[AsyncSession]:
        if self._sessionmaker is None:
            logger.error("sessionmaker is not available!")
            raise HTTPException(status_code=404)

        session = self._sessionmaker()

        try:
            yield session
        except SQLAlchemyError as e:
            await session.rollback()
            logger.error(f"Session error could not be established {e}")
            raise HTTPException(status_code=404)


sessionmanager = DataSessionManager()


async def get_db_session():
    async with sessionmanager.session() as session:
        yield session

and the docker-compose file and docker file..


   services:
     app:
       build:
         context: .
         dockerfile: Dockerfile
       ports:
         - "8000:8000"
       depends_on:
         - db
       environment:
         - DATABASE_URL=postgresql://admin:adminpass@db:5432/swifty_logs

     db:
       image: postgres:13
       environment:
         - POSTGRES_USER=admin
         - POSTGRES_PASSWORD=adminpass
         - POSTGRES_DB=swifty_logs
       volumes:
         - postgres_data:/var/lib/postgresql/data

   volumes:
     postgres_data:

# Use an official Python runtime as a parent image
FROM python:latest

# Set the working directory in the container
WORKDIR /app

# Copy the requirements file into the container
COPY ./requirements.txt  /app
# Install any dependencies specified in requirements.txt
RUN pip install --no-cache-dir --default-timeout=100 -r requirements.txt

# Copy the rest of the application code into the container
COPY . /app

# Expose the port FastAPI will run on
EXPOSE 8000

# Command to run the application with uvicorn
CMD  ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

after installing psycopg2…

app-1  | sqlalchemy.exc.InvalidRequestError: The asyncio extension requires an async driver to be used. The loaded 'psycopg2' is not async.

Answers

- Dunes
- September 26, 2024 at 9:31 am
- 0 votes
0
You need to install an async driver for PostgreSQL, and tell SQLAlchemy to use it.

asyncpg is one such driver. So after adding to your requirements file and installing it, you then have to tell SQLAlchemy to use it. You do this by modifying the database URL. ie.
```
DATABASE_URL=postgresql+asyncpg://admin:adminpass@db:5432/swifty_logs
                       ^^^^^^^^
                       | new bit           
```
SQLAlchemy docs on using different drivers for PostgreSQL — https://docs.sqlalchemy.org/en/20/core/engines.html#postgresql
Login or Signup to reply.

- BenyaminJafari
- September 26, 2024 at 9:56 am
- 0 votes
0
The error refers to the missing psycopg2 PostgreSQL driver. However, installing it is not the solution as the psycopg2 is a synchronous driver whereas you would need an async one like asyncpg. In this way, after installing it via pip you also need to update the database path within the docker-compose as follows:
```
DATABASE_URL=postgresql+asyncpg://admin:adminpass@db:5432/swifty_logs
```
Login or Signup to reply.

Please signup or login to give your own answer.

Click here to cancel reply.

Debian – Docker-Compose failing to Build Container when using postgresql

Answers