Protecting Our Server Against AI Scrapers: The Story of Anubis

You are seeing this message because the administrator of our website has taken proactive steps to protect us against the threat of aggressive AI companies scraping our content. This measure, known as Anubis, is a compromise that requires a bit of patience from our users.

Anubis uses a Proof-of-Work scheme similar to Hashcash, a proposed proof-of-work scheme designed to reduce email spam. At individual scales, the additional load is negligible, but when faced with mass scraper attacks, it becomes a significant hurdle for malicious actors. The ultimate goal of Anubis is not just to block scrapers, but also to gather intelligence on headless browsers – the tools used by many AI companies to scrape websites without being detected.

The design behind Anubis is clever: at a personal level, it's barely noticeable; however, when faced with an onslaught of scraper requests, it adds up quickly. This makes scraping much more expensive and time-consuming, discouraging malicious actors from targeting our site. However, this is not just about protection – it's also about giving us the opportunity to develop a better solution.

Unfortunately, Anubis requires modern JavaScript features that some plugins like JShelter may disable. For those who have been affected by these restrictions, please consider disabling JShelter or other similar plugins on our domain. This will ensure you can access our content without any issues.

What's Behind the Challenge?

The version of Anubis we're running is 1.20.0. We understand that this may seem like a hurdle, but rest assured it's necessary due to changes in the way website hosting has evolved in recent years. AI companies have altered the social contract, making traditional no-JS solutions inadequate. As such, we need to rely on more advanced methods like JavaScript to safeguard our content.

This is an ongoing work-in-progress, and we appreciate your patience as we strive to develop a better solution that doesn't require enabling JavaScript. Until then, please know that you're helping us navigate this new landscape by cooperating with the measures in place.