Merge branch 'ly/changed-path-traversal-with-magic-pathspec': The Battle Against AI Scraping

As you attempt to access this website, you're met with an unexpected obstacle: a protective measure designed to shield it against the relentless tide of AI-powered web scrapers. This is Anubis, a clever hack that seeks to level the playing field between legitimate users and malicious actors.

Anubis is a Proof-of-Work scheme that leverages concepts from Hashcash, a proposed proof-of-work scheme aimed at curbing email spam. While this mechanism might seem trivial at an individual scale, its impact becomes significant when applied to large-scale scraping operations. In essence, Anubis makes it more expensive and time-consuming for AI companies to scrape websites, effectively creating a barrier that slows them down.

But Anubis is not a foolproof solution, nor was it intended to be. Its primary purpose is to serve as a placeholder, buying valuable time for developers to focus on more advanced techniques – such as fingerprinting and identifying headless browsers – which are better equipped to distinguish between legitimate users and malicious actors.

However, this convenience comes with a caveat: Anubis relies on modern JavaScript features that plugins like JShelter can disable. If you're using such plugins, be sure to disable them for this website, as enabling JavaScript is currently the only way to bypass the challenge.

The current version of Anubis on this site is 1.21.3, and while a no-JS solution is being worked on, it remains a work-in-progress. In the meantime, users are required to enable JavaScript to progress through the challenge. This shift in the social contract around website hosting highlights the evolving nature of online security threats.

By understanding the purpose and limitations of Anubis, you're not only better equipped to navigate this website but also contributing to a broader conversation about web scraping, AI-powered security measures, and the ongoing struggle for digital safety.