skip to Main Content

I need to scrape a website that is js rendered. I found this nice working library requests_html that does the job. After using ‘pip install requests_html’ the following code will get the job done:

from requests_html import HTMLSession

url = examplesite

session = HTMLSession()
r = session.get(url)
r.html.render(sleep=1)
print(r.html.html)

The first time that it runs it will install chromium in order to render the url. However, when I try to use this code in an alpine dockerfile I get the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/root/.local/share/pyppeteer/local-chromium/588429/chrome-linux/chrome': '/root/.local/share/pyppeteer/local-chromium/588429/chrome-linux/chrome'

This is probably the case because the root folder is not present in the dockerfile. So how do I install chromium in a docker container? Also, I am not limited to this library, so if there are better ones to use that can work in a docker container, please let me know.

I already tried the following, but it didn’t work:

FROM python:3.7-alpine3.13

RUN apk add --no-cache  chromium --repository=http://dl-cdn.alpinelinux.org/alpine/v3.10/main

2

Answers


  1. Chosen as BEST ANSWER

    I fixed it using a different container:

    FROM python:3.8.13-slim-buster
    
    RUN apt-get update
    RUN apt install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
    

  2. If you want to still use alipne Docker image without any error, add these lines to your Dockerfile.

    FROM python:3.7-alpine3.13
    
    RUN apk add --no-cache chromium
    ENV PUPPETEER_EXECUTABLE_PATH /usr/bin/chromium-browser
    

    To avoid FileNotFoundError on the alpine version, manual selection of the chromium-browser is required for the Puppeteer.

    This is similar node.js example: Docker (node:8.15-alpine) + Chromium + Karma unit tests not working

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search