Protecting Websites from AI Scrapers: The Anubis Solution

Anubis is a Proof-of-Work scheme that was inspired by Hashcash, a proposed method for reducing email spam. At an individual level, the additional load caused by Anubis may seem insignificant, but when multiple AI companies engage in mass scraping, it becomes a significant hurdle. This added expense discourages malicious actors from attempting to scrape websites excessively.

The primary purpose of Anubis is not to eliminate AI scrapers entirely, but rather to create a "good enough" placeholder solution that allows website administrators more time to develop and implement more sophisticated security measures, such as fingerprinting and identifying headless browsers. By doing so, the challenge proof-of-work page can be made less accessible to legitimate users, thereby reducing the risk of downtime for websites.

However, Anubis does come with some limitations. It requires the use of modern JavaScript features that plugins like JShelter may disable. As a result, it is essential to disable these plugins or other similar ones when visiting this website. The current version of Anubis used by this website is 1.21.3.

Another important note is that users must enable JavaScript to proceed past the challenge. This requirement stems from the changing social contract around how website hosting works, with AI companies redefining what it means for a website to be accessible and usable. While a no-JS solution is still being developed, Anubis remains an essential tool in the ongoing effort to protect websites against AI scrapers.

Understanding the Trade-Offs of Anubis

Anubis represents a necessary compromise between security and user experience. By enabling this measure, website administrators can help prevent AI companies from exploiting their resources without compromising the usability of their platforms for legitimate users. However, it also means that some users may need to adjust their browser settings or disable certain plugins to access the content.

As technology continues to evolve, so too will the measures taken to safeguard websites against AI scrapers. While Anubis is an effective solution in the short term, its limitations serve as a reminder that ongoing innovation and collaboration between website administrators, security experts, and tech companies are crucial for creating a safer online environment.