Question 300908 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – Sudden "Protocol error (Page.navigate): Target closed" in Puppeteer-Lambda Integration

I’m encountering a "Protocol error (Page.navigate): Target closed" with Puppeteer in AWS Lambda. This issue started in our production environment after Dec 5th, 12:00pm EST, without any recent changes to the code.

The error was not present initially in our staging environment. However, after re-running our last GitHub action (with no code changes), the same error started occurring in staging, mirroring the production issue.

I’m perplexed by this sudden occurrence of the error in both environments without any code modifications. What could be causing this issue? Are there any specific logs, settings, or configurations within AWS Lambda or Puppeteer that I should examine? Any insights or suggestions on troubleshooting steps would be greatly appreciated.

Tried changing node version, updating chromium, changing args, using puppeteer core.

Code snippet:

import puppeteer from 'puppeteer-extra';
import stealthPlugin from 'puppeteer-extra-plugin-stealth';

const stealth = stealthPlugin();
puppeteer.use(stealth);

(async () => {
    try {
        const browser = await puppeteer.launch({
            headless: true, // Set to false if you need a visible browser window
            args: ['--no-sandbox', '--disable-setuid-sandbox']
        });

        const page = await browser.newPage();

        const userAgent = 'Mozilla/5.0 (Linux; Android 7.0; LGL84VL Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36';
        await page.setUserAgent(userAgent);

        const url = 'https://example.com'; // Replace with the target URL
        await page.goto(url, { waitUntil: 'networkidle2' });

        // Additional actions on the page...

        await browser.close();
    } catch (error) {
        console.error('Error occurred:', error);
    }
})();

Package.json

{
  "name": "scrp",
  "version": "2.4.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node build/main.js",
    "build": "node node_modules/typescript/bin/tsc",
    "postbuild": "cp -r src/assets build/",
    "dev": "node node_modules/nodemon/bin/nodemon.js --files src/main.ts",
    "install": "node node_modules/rimraf/bin.js node_modules/mongoose/index.d.ts"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@google-cloud/vision": "^3.0.1",
    "@sentry/node": "^6.19.7",
    "@sentry/tracing": "^6.19.7",
    "@socket.io/redis-adapter": "^7.1.0",
    "@sparticuz/chromium": "^109.0.5",
    "aws-sdk": "^2.1034.0",
    "axios": "^0.21.1",
    "bcryptjs": "^2.4.3",
    "bullmq": "^1.54.3",
    "cors": "^2.8.5",
    "csvtojson": "^2.0.10",
    "dotenv": "^10.0.0",
    "express": "^4.17.1",
    "helmet": "^4.6.0",
    "ioredis": "^4.28.0",
    "jsonwebtoken": "^8.5.1",
    "moment": "^2.29.4",
    "mongoose": "^6.3.1",
    "mongoose-lean-virtuals": "^0.9.0",
    "multer": "^1.4.3",
    "nodemailer": "^6.4.16",
    "object-hash": "^2.2.0",
    "okrabyte": "^0.1.1",
    "pending-xhr-puppeteer": "^2.3.3",
    "phone": "^2.4.22",
    "puppeteer": "19.4",
    "puppeteer-extra": "^3.3.6",
    "puppeteer-extra-plugin-recaptcha": "^3.6.8",
    "puppeteer-extra-plugin-stealth": "^2.11.2",
    "redis": "^4.0.6",
    "reflect-metadata": "^0.1.13",
    "rimraf": "^3.0.2",
    "socket.io": "^4.0.0",
    "socket.io-emitter": "^3.2.0",
    "spinnies": "^0.5.1",
    "winston": "^3.7.2"
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.114",
    "@types/aws-sdk": "^2.7.0",
    "@types/bcryptjs": "^2.4.2",
    "@types/cors": "^2.8.8",
    "@types/express": "^4.17.13",
    "@types/ioredis": "^4.28.1",
    "@types/nodemailer": "^6.4.0",
    "@types/object-hash": "^2.2.1",
    "@types/phone": "^2.4.1",
    "@types/rimraf": "^3.0.2",
    "nodemon": "^2.0.12",
    "ts-node": "^10.9.1",
    "typescript": "^4.9.3"
  }
}
Tags:

3

Answers


  1. 0

    @Gio G, I’ve face exactly the same problem since yesterday in one of my three AWS Lambda apps that run chromium from @sparticuz/chromium and puppeteer-core. So far I tried:

    • different versions of chrome and puppeteer
    • deploying a docker image of another lambda function that works well, but after deploying to that failing app it was still failing
    • rebuilding the lambda function from scratch, still receiving errors.
    • increase lambda memory from 1,5GB to 4GB

    It looks like it’s related to a lambda function backend not puppeteer or chromium directly. In local environment (using the same docker container) everything works great. Any ideas what else to check and how to fix it?

    Login or Signup to reply.
  2. 0

    I got the very same error and found no proper working solution as of this writing. The culprit is the npm install inside Dockerfile as it alters the package-lock.json file with new versions of puppeteer and its dependencies — which happens to be faulty. Changing it to npm ci might work but not on my end for some reason (you can try it on your end).

    The old image is working, so what I did is a mere workaround just keep things running – Alter the existing image.

    # Make sure to backup your working image by creating a new dockerfile with
FROM existing_image

# Once copied, run the existing image
docker run -dt existing_image 

# Check if it's running and grab it's container id
docker ps
# CONTAINER ID   IMAGE            COMMAND   CREATED              STATUS              
# 643bf857fceb   existing-image   "R"       6 minutes ago        Up 6 minutes

# Shell into it
docker exec -it 643bf857fceb bash

# delete outdated src (note that this is inside the existing image)
rm -rf src

# Apply code changes (note do the command from other terminal)
docker cp src 643bf857fceb:/var/task/

# Open another terminal tab/window, and save the running container you modified
docker commit 643bf857fceb my-new-image

# Inspect to ensure it saved correctly
docker image ls
# REPOSITORY           TAG       IMAGE ID       CREATED         SIZE
# existing-image       latest    a7dde5d84fe5   7 minutes ago   888MB
# my-new-image         latest    d57fd15d5a95   2 minutes ago   888MB

# Push my-new-image to aws container registry
docker push xxxx/my-new-image

    Note that I don’t have any dependency update, it is a pure src code logic so this is a suited temporary solution for me.

    Hope this helps.

    Login or Signup to reply.
  3. 0

    Same exact Error of @Gio,
    dependencies i have that works locally and in an aws server but not lambda function:

    "puppeteer": "^21.6.0",
"puppeteer-extra": "^3.3.6",
"puppeteer-extra-plugin-recaptcha": "^3.6.8",
"puppeteer-extra-plugin-stealth": "^2.11.2",

    I have tried moving to puppeteer-core but did not work either.
    Will try the image suggestion.
    Line that breaks is:

    import puppeteerExtra from 'puppeteer-extra';
var browser = await puppeteerExtra.launch({ ...options });
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search