Puppeteer v14.4.0 (and higher) broken when using puppeteer-extras #78

corford · 2022-06-30T21:43:36Z

browserController.newPage() fails with "Cannot read private member from an object whose class did not declare it" when used with puppeteer-core v14.4.0+ and puppeteer-extra v3.3.0 (with puppeteer-extra-plugin-stealth v2.10.1).

Seems to be a browser-pool issue (possibly Typescript related?), most likely caused by this puppeteer commit: puppeteer/puppeteer@v14.3.0...v14.4.0 (a lot of class fields were made private in that commit).

FYI, I have no issue running puppeteer natively without browser-pool e.g. this works with puppeteer-core 14.4.0 (and higher):

import vanillaPuppeteer from 'puppeteer-core';
import { addExtra } from 'puppeteer-extra';
import Stealth from 'puppeteer-extra-plugin-stealth';
import UserAgentOverride from 'puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js'; // eslint-disable-line max-len
import WebGLVendor from 'puppeteer-extra-plugin-stealth/evasions/webgl.vendor/index.js';
import UserPreferences from 'puppeteer-extra-plugin-user-preferences/index.js';

const stealth = Stealth();
stealth.enabledEvasions.delete('user-agent-override');
stealth.enabledEvasions.delete('webgl.vendor');

const prefs = UserPreferences({
  userPrefs: {
    intl: {
      accept_languages: 'en-US',
    },
    webrtc: {
      ip_handling_policy: 'disable_non_proxied_udp',
      multiple_routes_enabled: false,
      nonproxied_udp_enabled: false,
    },
  },
});

const ua = UserAgentOverride({
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36',
  locale: 'en-US',
  maskLinux: true,
});

const webgl = WebGLVendor({
  vendor: 'Google Inc. (Intel)',
  renderer: 'ANGLE (Intel, Intel(R) UHD Graphics Direct3D11 vs_5_0 ps_5_0, D3D11-27.20.100.8984)',
});

const puppeteer = addExtra(vanillaPuppeteer);
puppeteer.use(stealth);
puppeteer.use(prefs);
puppeteer.use(ua);
puppeteer.use(webgl);

puppeteer.launch({ executablePath: '/usr/bin/google-chrome' }).then(async browser => {
  const page = await browser.newPage();
  await page.setViewport({ width: 800, height: 600 });

  console.log(`Testing the stealth plugin..`);
  await page.goto('https://bot.sannysoft.com');
  await page.waitForTimeout(5000);
  await page.screenshot({ path: 'stealth.png', fullPage: true });

  console.log(`All done, check the screenshots. ✨`);
  await browser.close();
});

The text was updated successfully, but these errors were encountered:

B4nan · 2022-06-30T21:47:48Z

cc @vladfrangu this is what I was talking about today

B4nan · 2022-06-30T21:52:56Z

(possibly Typescript related?),

Seeing that commit and native private fields, I doubt this is about TS. We are apparently using some API that is no longer available (I don't believe there is a way around native private fields).

corford · 2022-06-30T21:54:54Z

We are apparently using some API that is no longer available

Thinking about it, yep that makes more sense than TS

corford · 2022-06-30T21:58:34Z

The bit that isn't super clear to me is why it only manifests when wrapped with puppeteer-extra

B4nan · 2022-06-30T22:00:36Z

berstend/puppeteer-extra#653

B4nan · 2022-07-01T08:20:24Z

I guess we can close this, right? You are apparently using outdated version of puppeteer-extra, the PR I mentioned is in v2.10 and you are on v2.3.1

corford · 2022-07-01T08:39:23Z

Sorry @B4nan created the ticket late last night (too sleepy). I'm actually on 2.10.1 of puppeteer-extra-plugin-stealth (and 2.3.1 of puppeteer-extra-plugin-user-preferences)

I've updated the ticket description with the correct version

corford · 2022-07-01T08:40:25Z

So I have those latest changes to pages._client() yet still have the issue with puppeteer v14.4.40+ and browser-pool

B4nan · 2022-07-01T08:54:02Z

Can you give us some reproduction? Your snippet is the working one, right? So provide failing one instead, that's more important than showing how to work it around.

corford · 2022-07-01T09:49:41Z

Ok, here's a minimal reproduction that triggers the same "Cannot read private member from an object whose class did not declare it" error.

Note: all of below is with puppeteer-core: ^14.4.0, puppeteer-extra: ^3.3.0, puppeteer-extra-plugin: ^3.2.0, puppeteer-extra-plugin-stealth: ^2.10.1, puppeteer-extra-plugin-user-preferences: ^2.3.1 and apify: ^2.3.2

Using puppeteer directly with stealth plugin (this works, no error)

import vanillaPuppeteer from 'puppeteer-core';
import { addExtra } from 'puppeteer-extra';
import Stealth from 'puppeteer-extra-plugin-stealth';
import UserAgentOverride from 'puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js'; // eslint-disable-line max-len
import WebGLVendor from 'puppeteer-extra-plugin-stealth/evasions/webgl.vendor/index.js';
import UserPreferences from 'puppeteer-extra-plugin-user-preferences/index.js';

const stealth = Stealth();
stealth.enabledEvasions.delete('user-agent-override');
stealth.enabledEvasions.delete('webgl.vendor');

const prefs = UserPreferences({
  userPrefs: {
    intl: {
      accept_languages: 'en-US',
    },
    webrtc: {
      ip_handling_policy: 'disable_non_proxied_udp',
      multiple_routes_enabled: false,
      nonproxied_udp_enabled: false,
    },
  },
});

const ua = UserAgentOverride({
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36',
  locale: 'en-US',
  maskLinux: true,
});

const webgl = WebGLVendor({
  vendor: 'Google Inc. (Intel)',
  renderer: 'ANGLE (Intel, Intel(R) UHD Graphics Direct3D11 vs_5_0 ps_5_0, D3D11-27.20.100.8984)',
});

const puppeteer = addExtra(vanillaPuppeteer);
puppeteer.use(stealth);
puppeteer.use(prefs);
puppeteer.use(ua);
puppeteer.use(webgl);

(async () => {
  const browser = await puppeteer.launch({ executablePath: '/usr/bin/google-chrome' });
  const page = await browser.newPage();
  await page.setViewport({ width: 800, height: 600 });

  console.log(`Testing the stealth plugin..`);
  await page.goto('https://bot.sannysoft.com');
  await page.waitForTimeout(5000);
  await page.screenshot({ path: 'stealth.png', fullPage: true });

  console.log(`All done, check the screenshots. ✨`);
  await browser.close();
})();

Using Apify.launchPuppeteer with stealth plugin (this triggers the error when browser closes)

import Apify from 'apify';
import vanillaPuppeteer from 'puppeteer-core';
import { addExtra } from 'puppeteer-extra';
import Stealth from 'puppeteer-extra-plugin-stealth';
import UserAgentOverride from 'puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js'; // eslint-disable-line max-len
import WebGLVendor from 'puppeteer-extra-plugin-stealth/evasions/webgl.vendor/index.js';
import UserPreferences from 'puppeteer-extra-plugin-user-preferences/index.js';

const stealth = Stealth();
stealth.enabledEvasions.delete('user-agent-override');
stealth.enabledEvasions.delete('webgl.vendor');

const prefs = UserPreferences({
  userPrefs: {
    intl: {
      accept_languages: 'en-US',
    },
    webrtc: {
      ip_handling_policy: 'disable_non_proxied_udp',
      multiple_routes_enabled: false,
      nonproxied_udp_enabled: false,
    },
  },
});

const ua = UserAgentOverride({
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36',
  locale: 'en-US',
  maskLinux: true,
});

const webgl = WebGLVendor({
  vendor: 'Google Inc. (Intel)',
  renderer: 'ANGLE (Intel, Intel(R) UHD Graphics Direct3D11 vs_5_0 ps_5_0, D3D11-27.20.100.8984)',
});

const puppeteer = addExtra(vanillaPuppeteer);
puppeteer.use(stealth);
puppeteer.use(prefs);
puppeteer.use(ua);
puppeteer.use(webgl);

(async () => {
  const browser = await Apify.launchPuppeteer({
    useChrome: false,
    stealth: false,
    launchOptions: {
      executablePath: '/usr/bin/google-chrome',
    },
    launcher: puppeteer,
  });

  const page = await browser.newPage();
  await page.setViewport({ width: 800, height: 600 });

  console.log(`Testing the stealth plugin..`);
  await page.goto('https://bot.sannysoft.com');
  await page.waitForTimeout(5000);
  await page.screenshot({ path: 'stealth.png', fullPage: true });

  console.log(`All done, check the screenshots. ✨`);
  await browser.close();
})();

Using Apify.launchPuppeteer without stealth plugin (this works, no errors)

import Apify from 'apify';

(async () => {
  const browser = await Apify.launchPuppeteer({
    useChrome: false,
    stealth: false,
    launchOptions: {
      executablePath: '/usr/bin/google-chrome',
    },
  });

  const page = await browser.newPage();
  await page.setViewport({ width: 800, height: 600 });

  console.log(`Testing without stealth plugin..`);
  await page.goto('https://bot.sannysoft.com');
  await page.waitForTimeout(5000);
  await page.screenshot({ path: 'stealth.png', fullPage: true });

  console.log(`All done, check the screenshots. ✨`);
  await browser.close();
})();

corford · 2022-07-01T10:59:55Z

Are you able to reproduce based on above or is this issue something local to my setup?

B4nan · 2022-07-01T11:06:02Z

I am able to see the same (or similar) in our tests even without that puppeteer-extra package - and that is the issue we primarily need to solve as we will be releasing Crawlee (successor to Apify SDK) during next week or two. I hope it will be the same problem as yours, still can't promise we will backport this to v2 (it might end up being a breaking fix).

corford · 2022-07-01T11:10:35Z

Ok, thanks for the extra context and info. Looking forward to kicking the tyres of Crawlee once it's out (and praying the work to port our internal framework - which is currently built on top of Apify SDK - wont be too enormous!).

B4nan · 2022-07-01T13:40:26Z

Some observations:

they use native private props, but TS is compiling it to lower level, so in fact its a TS helper called __classPrivateFieldGet (afaik the behaviour is the same as if there would be real native private props, so does not really matter)
we do wrap the Browser class with a Proxy here, and that is apparently causing the issues
calling public methods on the proxy results in this validation being triggered, as this is the proxy instance and it cannot access the private property as it belongs to the Browser class, not to Proxy

B4nan · 2022-07-01T14:27:55Z

Right, looks like I found a way to get around it, your repro is passing with the changes from the linked PR. There are probably more things to handle this way but for your repro it was enough to handle close and createIncognitoBrowserContext methods. Will first make the PR pass all the crawlee tests before backporting it here, but I dont see any roadblocks, won't be breaking.

corford · 2022-07-01T14:45:42Z

Nice one @B4nan 💪

B4nan mentioned this issue Jul 1, 2022

fix: support puppeteer 14.4+ apify/apify-ts#160

Merged

B4nan linked a pull request Jul 1, 2022 that will close this issue

fix: support puppeteer v14.4+ #79

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Puppeteer v14.4.0 (and higher) broken when using puppeteer-extras #78

Puppeteer v14.4.0 (and higher) broken when using puppeteer-extras #78

corford commented Jun 30, 2022 •

edited

Loading

B4nan commented Jun 30, 2022

B4nan commented Jun 30, 2022

corford commented Jun 30, 2022 •

edited

Loading

corford commented Jun 30, 2022 •

edited

Loading

B4nan commented Jun 30, 2022

B4nan commented Jul 1, 2022 •

edited

Loading

corford commented Jul 1, 2022 •

edited

Loading

corford commented Jul 1, 2022

B4nan commented Jul 1, 2022

corford commented Jul 1, 2022 •

edited

Loading

corford commented Jul 1, 2022

B4nan commented Jul 1, 2022

corford commented Jul 1, 2022

B4nan commented Jul 1, 2022

B4nan commented Jul 1, 2022

corford commented Jul 1, 2022

Puppeteer v14.4.0 (and higher) broken when using puppeteer-extras #78

Puppeteer v14.4.0 (and higher) broken when using puppeteer-extras #78

Comments

corford commented Jun 30, 2022 • edited Loading

B4nan commented Jun 30, 2022

B4nan commented Jun 30, 2022

corford commented Jun 30, 2022 • edited Loading

corford commented Jun 30, 2022 • edited Loading

B4nan commented Jun 30, 2022

B4nan commented Jul 1, 2022 • edited Loading

corford commented Jul 1, 2022 • edited Loading

corford commented Jul 1, 2022

B4nan commented Jul 1, 2022

corford commented Jul 1, 2022 • edited Loading

Using puppeteer directly with stealth plugin (this works, no error)

Using Apify.launchPuppeteer with stealth plugin (this triggers the error when browser closes)

Using Apify.launchPuppeteer without stealth plugin (this works, no errors)

corford commented Jul 1, 2022

B4nan commented Jul 1, 2022

corford commented Jul 1, 2022

B4nan commented Jul 1, 2022

B4nan commented Jul 1, 2022

corford commented Jul 1, 2022

corford commented Jun 30, 2022 •

edited

Loading

corford commented Jun 30, 2022 •

edited

Loading

corford commented Jun 30, 2022 •

edited

Loading

B4nan commented Jul 1, 2022 •

edited

Loading

corford commented Jul 1, 2022 •

edited

Loading

corford commented Jul 1, 2022 •

edited

Loading