Merge branch 'js/misc-fixes'
Have you found yourself struggling to access a website, only to be met with a frustrating message that says "you're seeing this because the administrator of this website has set up Anubis"? You're not alone. Many websites have been affected by the increasing scourge of AI companies aggressively scraping their content without permission. This not only harms the websites themselves but also makes resources inaccessible to everyone.
The culprit behind this issue is a system called Anubis, designed to protect servers from these malicious activities. While it may seem like a heavy-handed approach, Anubis serves as a compromise between security and user experience. It employs a Proof-of-Work scheme similar to Hashcash, which adds an extra layer of difficulty for AI companies trying to scrape websites en masse.
The idea behind Anubis is that at the individual level, the additional load it imposes is negligible. However, when mass scrapers try to exploit this loophole, it becomes a significant hurdle. This, in turn, makes scraping much more expensive and time-consuming for AI companies. But what's the real purpose of Anubis? It's not just about thwarting malicious activities – it's also a way to buy time for website administrators to focus on more sophisticated methods of identifying legitimate users.
One such method is fingerprinting, which involves analyzing how browsers render fonts and other visual elements. By doing so, website admins can create more targeted challenges that are less likely to be overcome by AI-powered bots. However, Anubis requires the use of modern JavaScript features – a catch that some plugins like JShelter will disable.
That's why users are being asked to disable JShelter or similar plugins when visiting websites protected by Anubis. The reason is simple: enabling JavaScript is necessary to overcome the challenge proof-of-work page presented by Anubis. This is a work-in-progress, as website administrators are working on developing no-JS solutions that can keep up with the evolving threat landscape.
In short, Anubis is a complex system designed to balance security and user experience in the face of an increasingly malicious online world. While it may seem like a cumbersome solution, its implementation serves as a reminder of the ever-evolving cat-and-mouse game between website administrators and AI-powered scrapers.