I’ve found about 15 different way to set up Docker to connect to an existing Postgre database. None of them work for me. It seems that I’m connecting to a database. It appears to be empty. This is the final output from docker-compose up –build:
db_1 |
db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
db_1 |
db_1 | 2020-05-22 21:32:42.213 UTC [1] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
db_1 | 2020-05-22 21:32:42.214 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2020-05-22 21:32:42.214 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2020-05-22 21:32:42.216 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2020-05-22 21:32:42.227 UTC [26] LOG: database system was shut down at 2020-05-22 21:32:32 UTC
db_1 | 2020-05-22 21:32:42.231 UTC [1] LOG: database system is ready to accept connections
web_1 | Watching for file changes with StatReloader
web_1 | Performing system checks...
web_1 |
web_1 | System check identified no issues (0 silenced).
web_1 |
web_1 | You have 38 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, base, contenttypes, sessions.
web_1 | Run 'python manage.py migrate' to apply them.
web_1 | May 22, 2020 - 21:32:44
web_1 | Django version 3.0.4, using settings 'merrittsecurity.settings'
web_1 | Starting development server at http://0.0.0.0:8000/
web_1 | Quit the server with CONTROL-C.
The fact that it says I have 38 unapplied migrations tells me it’s working with an empty database. My PyCharm Django project works fine outside of Docker and it worked fine in Docker with SQLite3 before I converted it.
The log entry: listening on IPv4 address “0.0.0.0”, port 5432 seems odd. Could this be the problem?
docker-compose.yml:
version: '3.7'
services:
db:
# image: keinos/sqlite3:latest
image: postgres
expose:
- 5432
environment:
- POSTGRES_DB=databsename
- POSTGRES_USER=me
- POSTGRES_PASSWORD=dbpassword
volumes:
- pgdata:/usr/local/var/postgresql/
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8005:8000"
depends_on:
- db
volumes:
pgdata: {}
I’ve replaced the database name, user, and password here. They normally contain the actual values, not variables. I got this particular construction from several examples on how to set it up.
Dockerfile:
FROM ubuntu:18.04
FROM python:3.7.0-alpine
# Set work directory
RUN mkdir /code
WORKDIR /code
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
ENV PYTHONMALLOC debug
# Install dependencies
# The next two lines were added to get postgre to build
RUN
apk add --no-cache postgresql-libs &&
apk add --no-cache --virtual .build-deps gcc musl-dev postgresql-dev
# The next line was added to get pillow to build
RUN apk add build-base python-dev py-pip jpeg-dev zlib-dev
#python3 -m pip install -r requirements.txt --no-cache-dir &&
#apk --purge del .build-deps
#RUN apk add build-base python-dev py-pip jpeg-dev zlib-dev
ENV LIBRARY_PATH=/lib:/usr/lib
COPY requirements.txt /code/
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
RUN apk --purge del .build-deps
# Copy django project
ADD . /code/
The pertinent section of setting.py:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'mydatabase',
'USER': os.getenv('POSTGRE_USER'),
'PASSWORD': os.getenv('POSTGRE_PASSWORD'),
'HOST': os.getenv('POSTGRE_HOST'),
'PORT': '5432',
}
}
The POSTGRE_… values are in my .env file. They work fine outside of Docker with the exception of POSTGRE_HOST. I normally set it to localhost. That causes an error so I replaced it with db as per an example from the web. The other values are the same as supplied in the docker-compose.yml file above. This is on the latest version of MacOS. Any idea what I’m doing wrong?
In response to questions
psql shows a number of databases that I’ve added.
The database has about about six tables beyond the ones created in the initial Django migrate. If I run my project outside of Docker, I have no trouble accessing the database.
When I run SHOW data_directory; in pqsl it responds: /usr/local/var/postgres
I installed postgre on MacOS using homebrew.
The postgre sever is running on MacOS, the same one where I execute the docker-compose.
As mentioned the POSTGRE_… are ones I created to link to the actual data in .env. The .env file contains the actual values. These same values are repeated in the docker-compose.yml. I’m not it’s necessary to have them twice.
2
Answers
A Fundamental Issue
I'm going about this the wrong way. I converted to using postgre by first installing postgreSQL using homebrew and then bringing up the sever running on MacOS. I reconfigured my Django app to link to that server and imported the application data from a sqlite3 database. The Django app is now happy talking to a postgre server running on MacOS.
I then altered my Docker files to use postgre in place of sqlite3. This is where the problem started. If I understand the process correctly, docker-compose is creating a new postgre server inside the image. With the volumes declaration, I'm trying to point it to data created by the postgre server running directly on MacOS described above. If I could get it to work, it could get ugly as there would be two servers using the same data.
From doing some additional research, the purpose of the volumes declaration it to give the Docker image postgre server a place to store its data, which can then be used the next time it comes up. I see two options. One is to create a standalone docker image containing postgre and then importing my data to it using pg-admin or the like. I can then use that image in my various docker Django images as my data source.
The other option would be to remove all mention of a database from my Django docker-compose.yml and let Django use the server running on MacOS. I haven't tried this so I'm not sure it will work. It also partially negates using Docker in the first place.
The host you should be connecting to is
db
. As that is the name you gave your database container. Your DSN in this instance would bepostgres://me:dbpassword@db:5432/databsename
Do not use
0.0.0.0
forPOSTGRE_HOST
.0.0.0.0
is a special value and has specific means depending on where it’s used. One use is that servers can say they want to listen to connections on0.0.0.0
, which means they want to listen on all IP4 addresses that the OS might have. For instance, a computer might sometimes have an IP address for its ethernet cable connection and another IP address for its WiFi connection. However,0.0.0.0
is not a valid IP address to connect to and is routed nowhere.See https://docs.docker.com/compose/networking/ for more details.