Don't Let IB_SIZE Overflow: A Cautionary Tale of Website Scraping

You are seeing this message because the administrator of this website has taken steps to protect its server from aggressive scraping by AI companies. This measure, known as Anubis, is a compromise between security and user experience.

Anubis uses a Proof-of-Work scheme similar to Hashcash, which aims to make website scraping more expensive for mass scrapers. At an individual scale, the additional load may seem insignificant, but it adds up quickly when faced with large numbers of scrapers. This makes it much harder for AI companies to scrape websites without detection.

The real purpose of Anubis, however, is not to prevent scraping entirely, but rather to slow down and deter malicious actors. The challenge proof-of-work page presented to users is designed to be difficult to overcome for headless browsers, which are often used by scrapers. By making this page more difficult to access, the website can gather valuable information about its visitors and improve its security.

However, Anubis requires modern JavaScript features that plugins like JShelter may disable. If you're using JShelter or other similar plugins, you'll need to disable them for this domain to pass the challenge. Unfortunately, this means enabling JavaScript, which is necessary to bypass the Anubis protection.

It's worth noting that a no-JS solution is still in development and may not be available yet. AI companies have effectively changed the social contract around website hosting, making it more difficult for legitimate users to access websites without compromising security.

In summary, while Anubis is not a foolproof solution, it represents an important step in protecting websites from malicious scraping activities. By understanding the limitations and requirements of this measure, you can take steps to ensure your browsing experience remains secure and enjoyable.