Merge branch 'am/sparse-index-name-hash-fix': The Double-Edged Sword of Anubis

As a user, you're likely to encounter an unexpected hurdle when trying to access certain websites. A message, similar to the one displayed here, is presented before the website's content can be loaded. This is not a typical error message, but rather an attempt by the website administrator to protect their server from Aggressive Scraping, a practice commonly employed by AI companies to extract valuable data from websites.

The administrator of this website has set up Anubis, a system designed to safeguard against these malicious activities. By using a Proof-of-Work scheme similar to Hashcash, Anubis adds an additional layer of security to the website's content. At the individual scale, this may seem like an insignificant burden, but for mass scrapers, it becomes exponentially more difficult and expensive.

"A hack whose real purpose is to give a 'good enough' placeholder solution," some might say. Indeed, Anubis serves as a compromise between security and accessibility. Its primary goal is not to prevent all scraping, but rather to discourage aggressive companies from exploiting the system. In doing so, it allows more time for fingerprinting and identifying headless browsers, such as those used by social media bots. This information can then be used to develop targeted solutions that protect legitimate users.

"Please disable JShelter or other similar plugins" is a warning message you may encounter, accompanied by the requirement to enable JavaScript in order to proceed. This is not an ideal solution, but it's a necessary compromise due to the evolving nature of social contract around website hosting. AI companies have redefined what it means to host a website, and this shift has forced websites to adapt.

While Anubis provides a temporary barrier between legitimate users and malicious actors, its limitations are well-documented. As such, a no-JS solution is still a work-in-progress. For now, users will need to navigate these challenges in order to access the content they seek.

Stay vigilant, as Anubis continues to evolve and adapt to new threats.