Javascript – Cannot use external function inside page.evaluate()

As discussed in my comment, the approach here seems rather convoluted. I’d caution against premature abstractions.

In general, once you need to add multiple conditions (switch and if) where there weren’t any before, you may be headed down the wrong path. These increase the cognitive complexity of the code. Complexity in a function can be acceptable if it reduces complexity for the caller, but if the contract for the function isn’t clear, then the abstraction may leak problems back to the caller.

Packing all of the logic into your dataMapper function breaches the single responsibility principle and makes it unmaintainable, because you’ll need to keep burdening it further with additional types of structures. The control flow within the function is already difficult to grasp and can’t be extended in any sensible way. The caller should be responsible for explicitly encoding the structure to be scraped, rather than trying to write an all-in-one function that can’t sensibly be written for these structures.

Another rule of thumb: if the factoring is difficult, then just keep the repetition. Or take a step back and try to write a different abstraction, either at a higher or lower level than the first attempt.

In this case, you might write a couple of higher-level abstractions $$evalMap and $text, which let you write your data mappers more cleanly. These abstractions just clear some of the syntax out of the way, but don’t attempt to generalize scraping different structures with conditions.

const puppeteer = require("puppeteer"); // ^22.7.1

const url = "https://www.ozone.bg/pazeli-2d-3d/nastolni-igri";

let browser;
(async () => {
  browser = await puppeteer.launch();
  const [page] = await browser.pages();
  const ua =
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36";
  await page.setUserAgent(ua);
  await page.setRequestInterception(true);
  const blockedResources = [
    "image",
    "fetch",
    "other",
    "ping",
    "stylesheet",
    "xhr",
  ];
  page.on("request", req => {
    if (
      !req.url().startsWith("https://www.ozone.bg") ||
      blockedResources.includes(req.resourceType())
    ) {
      req.abort();
    } else {
      req.continue();
    }
  });
  await page.goto(url, {waitUntil: "domcontentloaded"});

  await page.evaluate(
    "window.$text = (el, s) => el.querySelector(s)?.textContent.trim();"
  );

  const $$evalMap = async (sel, mapFn) => {
    await page.waitForSelector(sel);
    return page.$$eval(
      sel,
      (els, mapFn) => els.map(new Function(`return ${mapFn}`)()),
      mapFn.toString()
    );
  };

  const carouselData = await $$evalMap(".owl-item", el => ({
    title: $text(el, ".title"),
    img: el.querySelector("img").src,
    url: el.querySelector("a").href,
  }));

  const widgetData = await $$evalMap(".widget-box", el => ({
    title: el.querySelector("img").alt,
    img: el.querySelector("img").src,
    url: el.querySelector("a").href,
  }));

  const sliderData = await $$evalMap(
    ".item.slick-slide",
    el => ({
      title: $text(el, "h3"),
      subTitle: $text(el, "h6"),
      img: el.querySelector("img").src,
      url: el.querySelector("a").href,
    })
  );

  console.log(carouselData);
  console.log(widgetData);
  console.log(sliderData);
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close());

That said, there’s not that much harm in doing the work directly in $$evals:

// ... same as above ...
await page.goto(url, {waitUntil: "domcontentloaded"});

await page.evaluate(
  "window.$text = (el, s) => el.querySelector(s)?.textContent.trim();"
);

await page.waitForSelector(".owl-item");
const carouselData = await page.$$eval(".owl-item", els =>
  els.map(el => ({
    title: $text(el, ".title"),
    img: el.querySelector("img").src,
    url: el.querySelector("a").href,
  }))
);

const widgetData = await page.$$eval(".widget-box", els =>
  els.map(el => ({
    title: el.querySelector("img").alt,
    img: el.querySelector("img").src,
    url: el.querySelector("a").href,
  }))
);

const sliderData = await page.$$eval(
  ".item.slick-slide",
  els =>
    els.map(el => ({
      title: $text(el, "h3"),
      subTitle: $text(el, "h6"),
      img: el.querySelector("img").src,
      url: el.querySelector("a").href,
    }))
);
// ...

If TypeScript types are getting in the way, consider moving the querySelectors out to similar helper functions as was done with $text. The $$eval calls can also be moved out to individual functions for each type.

Summary and further remarks:

• Avoid premature abstractions.
• When factoring, stop if you’re introducing multiple conditions where there weren’t any before. switch/case/break is particularly nasty. If your abstractions are more verbose and hard to understand than the original repeated code, don’t do them, or try to find a different abstraction.
• When factoring, write it the verbose way first, then try to abstract away the similarities (but a bit of repetition is acceptable–be honest about what’s easier to read and maintain).
• as is discouraged in TS. Use it as little as possible in favor of variable types.
• Use $$eval all the time. It’s the most generally useful scraping function in Puppeteer, avoiding an ugly Array.from(document.querySelectorAll) or element handles. If Puppeteer’s locators API matures in the future, $$eval may be supplanted, but for now it’s the way to go.
• ?? undefined is unnecessary. If the left hand chained operator ?. fails, the expression evaluates to undefined anyway, so undefined ?? undefined is pointless.
• Generally speaking, you don’t need addScriptTag or exposeFunction. If you’re writing $text-like abstractions often, you can jQuery or something like that to simplify querying.