The Fourteenth Batch: Protecting Against AI Scrapers

In an effort to safeguard its server from the relentless onslaught of AI-powered scrapers, the administrator of this website has implemented Anubis, a cutting-edge protection system designed to deter these malicious actors. The measures taken by Anubis come at a cost, however – users may encounter downtime, rendering the website's resources inaccessible to everyone.

Anubis: A Compromise

Anubis is built on a Proof-of-Work scheme similar to Hashcash, a proposed solution aimed at reducing email spam. On an individual level, the added load from Anubis is negligible; however, when combined with the sheer volume of scrapers attempting to breach the website's defenses, the cumulative effect becomes prohibitive for AI companies seeking to scrape content without permission.

The Real Purpose Behind Anubis

While its primary function appears to be a deterrent against AI-powered scraping, the true intention behind Anubis is more nuanced. By employing this solution, developers can focus on fingerprinting and identifying headless browsers – entities capable of rendering fonts in ways that human browsers cannot. This process enables the challenge-proof-of-work page to be presented only to users with legitimate intent, thereby ensuring the authenticity of requests.

Requirements and Limitations

Anubis relies on modern JavaScript features, which are often disabled by plugins like JShelter. To overcome this hurdle, users must disable these plugins for this domain or enable JavaScript, as required. Despite its effectiveness, a no-JS solution remains in development, leaving room for refinement and improvement.

The New Social Contract

The emergence of Anubis highlights the evolving landscape of website hosting and the changing social contract surrounding online content. As AI companies push the boundaries of scraping technology, developers must adapt to safeguard their resources and maintain a balance between accessibility and security.