I’m trying to connect my python container which is a web scrape and then upload the data to the postgresql container (database).
This is the error I get from the python container. Everything else runs good except this. I use selenium to scrape some data and then upload it to my database. This works locally, but when I run the docker compose up there’s this problem.
This is the output from my postgresql container
The question is what is the cause of this and how can I solve it?
Thanks in advance
Python:
from selenium import webdriver
from selenium.webdriver.common.by import By
import psycopg2
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--window-size=1920,1080')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--disable-dev-shm-usage')
db_params = {
"host": "localhost",
"database": "postgres",
"user": "postgres",
"password": "root",
}
class Listing:
def __init__(self, title, location, image_url):
self.title = title
self.location = location
self.image_url = image_url
def __str__(self):
return f"Title: {self.title}, Location: {self.location}, Image Url: {self.image_url}"
driver = webdriver.Chrome(options=chrome_options)
listings = []
# first 500 listings = 20 listings per page * 25 pages
for page in range(1, 25):
... (removed some code for better readability)
# save data in listing instance
listing = Listing(title, location, image_url)
listings.append(listing)
try:
conn = psycopg2.connect(**db_params)
cur = conn.cursor()
for listing in listings:
cur.execute("""
INSERT INTO listings(title,location,imageurl)
VALUES(%s, %s, %s);
""", (listing.title, listing.location, listing.image_url))
conn.commit()
conn.close()
except Exception as e:
print(f"Error: {e}")
compose.yml
version: '3'
services:
frontend:
build:
context: client
ports:
- 3000:3000
depends_on:
- backend
backend:
build:
context: .
dockerfile: server/Dockerfile
ports:
- 3001:3001
depends_on:
- database
database:
image: postgres
restart: always
environment:
POSTGRES_PASSWORD: root
ports:
- 5432:5432
python:
build:
context: flatScraper
2
Answers
You are trying to connect to
localhost
(i.e. the same container) but your database is running in a different container. You need to connect to the hostdatabase
because that is the name of the container your database is running in.Note that it is possible to set a different host name for a container but your setup does not do that so the container name is the host name.
I’ll provide a simple solution, if the python code is in a container as your database, it needs to connect not by using localhost, but instead the hostname you created for the database.
Add the
hostname
property to each service you have in the docker-compose file, then in python change the host you’re trying to connect to, to the hostname you inserted like this:and then in python:
Also don’t paste an image of the error next time!