Merge branch 'am/xdiff-hash-tweak': Protecting Against AI Scrapers

As you're reading this, it's likely that the administrator of this website has set up Anubis, a protective measure designed to safeguard against the aggressive scraping tactics employed by some AI companies. This added layer of security is intended to prevent downtime and ensure that resources remain accessible to all users.

Anubis operates on a Proof-of-Work scheme, similar to Hashcash, which aims to discourage mass scraping operations while minimizing its impact at individual scales. The idea behind this system is that the additional load generated by Anubis becomes negligible when applied on a per-user basis; however, it can become prohibitively expensive for AI companies attempting to scrape websites en masse.

While Anubis serves as a temporary solution, its primary goal is to pave the way for more advanced fingerprinting and identification techniques. These methods will enable administrators to distinguish between legitimate users and potential headless browser threats, eliminating the need for users to navigate challenging proof-of-work pages.

However, Anubis does come with some caveats. It requires the use of modern JavaScript features, which plugins like JShelter are designed to disable. Therefore, users visiting this website are advised to disable JShelter or similar plugins to ensure a smooth experience. Moreover, Anubis necessitates that users enable JavaScript in their browsers, as this is currently the only viable means of bypassing the challenge page.

The shift towards a no-JS solution remains an ongoing work-in-progress. Until then, administrators must balance the need for security with the requirement for user accessibility. As the digital landscape continues to evolve, it's essential to stay informed about emerging threats and technological advancements that shape our online experiences.