Protecting Against AI Scrapers: The Anubis Solution
As you're reading this article, it's likely that your browser has flagged our site as potentially malicious due to the efforts of a clever adversary - an artificial intelligence (AI) company looking to scrape our website for its valuable content. But why is this happening? And what can we do about it?
The Problem with AI Scrapers
AI companies are using sophisticated algorithms and automated tools to scrape websites, often without permission or consent. This not only puts a strain on our server resources but also compromises the security and integrity of the content we work hard to create. To combat this issue, our administrator has set up Anubis - a cutting-edge solution designed to protect our site against these malicious actors.
What is Anubis?
Anubis is a Proof-of-Work scheme inspired by Hashcash, a proposed proof-of-work scheme for reducing email spam. In essence, it's a challenge that requires users to complete a simple task - in this case, viewing the image on our site - which verifies they're not an AI bot. This seemingly innocuous request adds a layer of security to our website, making it more difficult for scrapers to access.
How Does It Work?
At individual scales, the additional load from Anubis may seem negligible, but when aggregated across multiple scraper attempts, it becomes significant. This is because AI companies are using large-scale automation tools that can generate an enormous volume of requests in a short period. By making these requests more expensive and time-consuming, we're effectively deterring them from targeting our site.
Enabling JavaScript: A Necessary Evil
Unfortunately, Anubis requires the use of modern JavaScript features to function correctly. However, plugins like JShelter can disable this functionality, rendering our website inaccessible. We kindly request that you disable such plugins or enable JavaScript to proceed safely.
The Future of Web Security
While Anubis provides a temporary solution, we're aware that it's not a permanent fix. The social contract around web hosting has shifted with the rise of AI-powered scraping. We're working on developing a no-JS solution to address this issue and provide an alternative way forward for our users.
We appreciate your patience and cooperation as we navigate these complex security challenges. By supporting our efforts, you'll be helping us create a safer and more secure online environment for everyone.