skip to Main Content

Python container

I’m trying to connect my python container which is a web scrape and then upload the data to the postgresql container (database).

This is the error I get from the python container. Everything else runs good except this. I use selenium to scrape some data and then upload it to my database. This works locally, but when I run the docker compose up there’s this problem.

Postgresql container
This is the output from my postgresql container

The question is what is the cause of this and how can I solve it?

Thanks in advance

Python:

from selenium import webdriver
from selenium.webdriver.common.by import By
import psycopg2

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--window-size=1920,1080')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--disable-dev-shm-usage')

db_params = {
    "host": "localhost",
    "database": "postgres",
    "user": "postgres",
    "password": "root",

}


class Listing:
    def __init__(self, title, location, image_url):
        self.title = title
        self.location = location
        self.image_url = image_url

    def __str__(self):
        return f"Title: {self.title}, Location: {self.location}, Image Url: {self.image_url}"


driver = webdriver.Chrome(options=chrome_options)

listings = []

# first 500 listings = 20 listings per page * 25 pages
for page in range(1, 25):
        ... (removed some code for better readability)
        # save data in listing instance
        listing = Listing(title, location, image_url)

        listings.append(listing)

try:
    conn = psycopg2.connect(**db_params)
    cur = conn.cursor()

    for listing in listings:
        cur.execute("""
            INSERT INTO listings(title,location,imageurl) 
            VALUES(%s, %s, %s);
        """, (listing.title, listing.location, listing.image_url))

        conn.commit()

    conn.close()
except Exception as e:
    print(f"Error: {e}")

compose.yml

version: '3'
services:
  frontend:
    build: 
      context: client
    ports:
      - 3000:3000
    depends_on: 
      - backend

  backend:
    build: 
      context: .
      dockerfile: server/Dockerfile
    ports: 
      - 3001:3001
    depends_on: 
      - database

  database:
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: root
    ports:
      - 5432:5432

  python:
    build: 
      context: flatScraper

2

Answers


  1. You are trying to connect to localhost (i.e. the same container) but your database is running in a different container. You need to connect to the host database because that is the name of the container your database is running in.

    Note that it is possible to set a different host name for a container but your setup does not do that so the container name is the host name.

    Login or Signup to reply.
  2. I’ll provide a simple solution, if the python code is in a container as your database, it needs to connect not by using localhost, but instead the hostname you created for the database.

    Add the hostname property to each service you have in the docker-compose file, then in python change the host you’re trying to connect to, to the hostname you inserted like this:

    ...
    postgres:
        hostname: "database"
    ...
    

    and then in python:

    # ...
    db_params = {
        "host": "database",
        "database": "postgres",
        "user": "postgres",
        "password": "root",
    }
    # ...
    

    Also don’t paste an image of the error next time!

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search