Protecting Against AI Scrapers: The Rise of Anubis

In an effort to combat the growing threat of AI-powered website scrapers, the administrator of this website has implemented a new measure to safeguard its content. This change may cause some downtime for users, but it is essential to ensure that human eyes remain engaged with the site.

Anubis: A Compromise Solution

Anubis is a Proof-of-Work scheme designed to discourage AI scrapers from accessing this website. By adding an extra layer of complexity to the page, Anubis makes it more difficult for bots to scrape content without human interaction. This approach may seem like a hack, but its primary goal is to provide a temporary placeholder solution while researchers continue to develop more sophisticated methods to fingerprint and identify headless browsers.

A Brief History: Hashcash and Beyond

The concept of Anubis draws inspiration from the Hashcash system, proposed as a way to reduce email spam. While the original proposal was met with skepticism, its underlying idea has been refined to better suit the needs of modern web development. The Proof-of-Work scheme used by Anubis aims to make scraping more expensive for mass-scale operations, but not so burdensome that it becomes impractical at individual scales.

The Catch: Modern JavaScript Required

However, Anubis comes with a catch – users must enable JavaScript to navigate the challenge page. This requirement is a direct result of AI companies redefining the social contract around website hosting. While a no-JS solution is still in development, it is essential that users like you are aware of this compromise and take steps to ensure their browser plugins do not interfere with Anubis' functionality.

Disabling JShelter: A Necessary Evil

For those who rely on browser plugins like JShelter, disabling them or finding alternative solutions is necessary to bypass the challenge page. The administrator's primary concern is protecting human users from AI scrapers, and this temporary measure is an essential step in achieving that goal.

A New Normal?

While Anubis may seem like a stopgap solution, its implementation marks a shift in the way we approach website security. As researchers continue to develop more sophisticated methods to combat AI scrapers, it's likely that Anubis will become an integral part of the web development landscape. For now, users must adapt to this new normal and find ways to work around the limitations imposed by Anubis.