Using Bright Data's "Scraping Browser" to by-pass CAPTCHA's and other protection when monitoring pages

Stephen — Thu, 16 Nov 2023 16:39:50 +0000

Using Bright Data's "Scraping Browser" to by-pass CAPTCHA's and other protection when monitoring pages Stephen Thu, 11/16/2023 - 17:39

Topic

How-To

Proxies

For many websites - simply using a proxy is not enough, the website is using much more complex anti-robot software to detect the actual browser that is being used, not just the IP connection or headers (including common headers such as user-agent and others)

You need a better way to simulate a real browser - A scraping browser is different to a proxy, a proxy just tunnels your connection but the Browser still looks the same, with a scraping browsers you are also simulating a real browser hidden away in Bright Data's infrastructure.

As it turns out, many websites are probing much deeper into your browser - analysing the response from your 2D and 3D/GPU videocard is just one of the tactics they use.

Unfortunately for many headless Chrome sessions it means they stand out all too easy - their "fingerprint" is just too obvious, they dont have anything that resembles a real video card or other hardware attached.

The result is - you get pushed to enter a CAPTCHA or other similar anti-robot mechanism.

But there is a solution, whilst not guaranteed - it definitely helps a lot!

The clever people over at Bright Data have added a "Scraping Browser" to their offers, which more precisely simulates a real browser than just about anything you can try to run yourself.

(note: this functionality in changedetection.io will be released late November 2023, but you can try it now under the current master tag from our GitHub or dev tag from our Docker Hub)

More information about Bright Data's scraping browser can be found here.

Here's how to setup Bright Data's "Scraping Browser" with changedetection.io

Head on over to your Bright Data control panel - When you sign up using this link - https://brightdata.grsm.io/n0r16zf7eivq - BrightData will match any first deposit up to $150.

Once logged in - click on "Scraping Browser"

Now we will copy the special "Connection address" so changedetection.io knows how to find the Scraping Browser, click on Check out code and integration examples on the bottom right.

It's also worth considering here that you can add on extra residential proxies , residential proxies also greatly increase your chance of skipping past any potential CAPTCHA issues, but for now we will continue with the default setup.

Copy the text from the next page into your clipboard, it will start with wss://, be sure to not include any quotes or other text, copy the whole text which is marked in blue below.

Now for the fun part, jump over to your changedetection.io login and click on Settings > CAPTCHA & Proxies, scroll down to the "Extra Browsers" section, give your browser a name (in this case just BrightData Scraping Browser and paste the wss://... type URL into the Connection URL box

Now to use your new Scraping Browser with any website you are watching for changes, simply click Edit in your overview list and select the new browser

Choose the new browser Bright Data Scraping Browser that we setup in the previous step.

And there you have it - how to setup a scraping browser to get a much better success rate (say goodbye CAPTCHA!) when watching websites for changes.

Some extra tips -

When you setup additional Residential Proxies - Bright Data will present you with a new "wss://.." type connection URL, so you can then setup multiple proxy networks with multiple scraping browsers.

Sign up with Bright Data using https://brightdata.grsm.io/n0r16zf7eivq BrightData will match any first deposit up to $150

Troubleshooting

If you see the error "Overriding accept-language, user-agent headers forbidden" - You may need to enable "Custom headers" - In your Brightdata dashboard - under the Proxy Settings > Configuration > Advanced Settings > Custom headers & cookies this option needs to be activated. (changedetection.io may send its own custom User-Agent and other headers)

If you receive an error like "WebSocket error: wss://brd-customer..... 403 wrong_auth" then try whitelisting the IP of our server (or your local connection) from inside the BrightData control panel at the "Scraping Browser" settings - as an alternative to username+password (But still keep the username+password in the wss://.. connection URL)

Happy changedetecting!

How to - Bright Data Proxies and changedetection.io

Stephen — Wed, 02 Aug 2023 15:36:12 +0000

How to - Bright Data Proxies and changedetection.io Stephen Wed, 08/02/2023 - 17:36

Topic

How-To

Proxies

Using Bright Data proxies with changedetection.io is super beneficial for accessing web pages from different countries and bypassing blocking rules for several reasons:

Geographical Location: Proxies allow you to route your web requests through servers located in different countries. By doing so, you can effectively appear as if you are accessing the web from that specific country. This is useful for accessing geo-restricted content or services that are only available to users in certain regions.
Bypassing IP Restrictions: Some websites implement IP-based restrictions to limit access to their content or prevent excessive requests from the same IP address. Proxies help you overcome these restrictions by changing your IP address, so you can access the website as if you were coming from a different IP.
Overcoming Censorship: In regions with strict internet censorship, certain websites and online services may be blocked by the government or internet service providers. Proxies can help users bypass such censorship and access blocked content.
Anonymity and Privacy: Proxies provide an extra layer of anonymity by masking your original IP address. This can be valuable for users who want to protect their online privacy and avoid being tracked by websites or third parties.
Load Distribution: When accessing websites through proxies, the load is distributed across multiple servers. This can help prevent server overload and improve the overall performance and speed of web scraping or data collection activities.
Scalability: Proxies allow you to scale your web scraping or data collection efforts by making multiple requests from different IP addresses. This helps avoid getting blocked by websites that have rate-limiting or anti-scraping measures in place.
Legal and Ethical Compliance: Proxies can be used to access websites in a manner that is compliant with their terms of service and legal regulations. By rotating IP addresses and not overloading servers, you can ensure responsible data collection practices.

It's super easy to connect Bright Data proxies to your changedetection.io account or installation, head on over to https://brightdata.com/integration/changedetection to find out more!