Skip to main content

Using Bright Data's "Scraping Browser" to by-pass CAPTCHA's and other protection when monitoring pages

By Stephen, Updated 16 November, 2023
Topic

For many websites - simply using a proxy is not enough, the website is using much more complex anti-robot software to detect the actual browser that is being used, not just the IP connection or headers (including common headers such as user-agent and others)

You need a better way to simulate a real browser - A scraping browser is different to a proxy, a proxy just tunnels your connection but the Browser still looks the same, with a scraping browsers you are also simulating a real browser hidden away in Bright Data's infrastructure.

As it turns out, many websites are probing much deeper into your browser - analysing the response from your 2D and 3D/GPU videocard is just one of the tactics they use.

Unfortunately for many headless Chrome sessions it means they stand out all too easy - their "fingerprint" is just too obvious, they dont have anything that resembles a real video card or other hardware attached.

The result is - you get pushed to enter a CAPTCHA or other similar anti-robot mechanism.

But there is a solution, whilst not guaranteed - it definitely helps a lot! 

The clever people over at Bright Data have added a "Scraping Browser" to their offers, which more precisely simulates a real browser than just about anything you can try to run yourself.

(note: this functionality in changedetection.io will be released late November 2023, but you can try it now under the current master tag from our GitHub or dev tag from our Docker Hub)
 

More information about Bright Data's scraping browser can be found here.

 

Here's how to setup Bright Data's "Scraping Browser" with changedetection.io

 

Head on over to your Bright Data control panel - When you sign up using this link -  https://brightdata.grsm.io/n0r16zf7eivq - BrightData will match any first deposit up to $150.

Once logged in - click on "Scraping Browser"

 

The Bright Data control panel - where to find the Scraping Browser link

 

Now we will copy the special "Connection address" so changedetection.io knows how to find the Scraping Browser, click on Check out code and integration examples on the bottom right.

It's also worth considering here that you can add on extra residential proxies , residential proxies also greatly increase your chance of skipping past any potential CAPTCHA issues, but for now we will continue with the default setup.


The Bright Data control panel - where to find the Scraping Browser link

 

Copy the text from the next page into your clipboard, it will start with wss://, be sure to not include any quotes or other text, copy the whole text which is marked in blue below.

 

The Bright Data control panel - where to find the Scraping Browser wss:// link for changedetection

 

Now for the fun part, jump over to  your changedetection.io login and click on Settings > CAPTCHA & Proxies, scroll down to the "Extra Browsers" section, give your browser a name (in this case just BrightData Scraping Browser and paste the wss://... type URL into the Connection URL box

 

Adding a scraping browser to changedetection.io - step 1 adding the browser link

 

Now to use your new Scraping Browser with any website you are watching for changes, simply click Edit in your overview list and select the new browser 

 

list of websites with changes overview

 

Choose the new browser Bright Data Scraping Browser that we setup in the previous step.

 

Setting a single website to use a particular scraping browsers

 

And there you have it - how to setup a scraping browser to get a much better success rate (say goodbye CAPTCHA!) when watching websites for changes.

 

Some extra tips -

When you setup additional Residential Proxies - Bright Data will present you with a new "wss://.." type connection URL, so you can then setup multiple proxy networks with multiple scraping browsers.

Sign up with Bright Data using https://brightdata.grsm.io/n0r16zf7eivq BrightData will match any first deposit up to $150

 

Troubleshooting 

If you receive an error like "WebSocket error: wss://brd-customer..... 403 wrong_auth" then try whitelisting the IP of our server (or your local connection) from inside the BrightData control panel at the "Scraping Browser" settings - as an alternative to username+password (But still keep the username+password in the wss://.. connection URL)

 

Example of adding a white-list IP access in BrightData for the scraping browser

Happy changedetecting!