Merge branch 'jc/test-hashmap-is-still-here': The Growing Threat of Bot Scrapers and the Evolution of Website Protection

As we navigate the digital landscape, it's essential to acknowledge the increasing presence of bot scrapers and their impact on website security. These malicious entities have become a significant threat, with AI companies aggressively scraping websites in search of valuable data. In response, administrators have resorted to implementing Anubis, a Proof-of-Work scheme designed to protect servers from these scourges.

Anubis: A Compromise for Website Protection

Anubis is a measure taken by website administrators to safeguard their resources against bot scrapers. This system uses a Proof-of-Work scheme similar to Hashcash, which has been proposed as a solution to reduce email spam. While individual users may not notice the additional load imposed by Anubis, mass scraper levels can make it significantly more expensive for these entities.

The Purpose of Anubis: A Placeholder Solution

The primary intention behind Anubis is to provide a "good enough" placeholder solution that allows administrators to spend more time on fingerprinting and identifying headless browsers. By targeting users who are more likely to be legitimate, the challenge proof of work page can be presented without hindering genuine website interactions.

Prerequisites for Navigating the Anubis Challenge

To overcome the Anubis challenge, users must enable JavaScript, which is currently required due to the evolving social contract around website hosting. A no-JS solution is still in development and may not be available at this time. Furthermore, plugins like JShelter, which can disable modern JavaScript features, are not recommended for accessing websites protected by Anubis.

Anubis Version 1.20.0: The Current State

This website is currently running Anubis version 1.20.0. It's essential to note that users must enable JavaScript to bypass the challenge and access the site. While this measure may seem inconvenient, it represents a necessary step in safeguarding against bot scrapers and protecting the integrity of online resources.

Enabling Accessibility: A Requirement for Website Protection

To ensure seamless interactions with websites protected by Anubis, users are advised to disable JShelter or other plugins that can interfere with modern JavaScript features. By taking these precautions, users can help mitigate the impact of bot scrapers and contribute to a safer online environment.