Merge branch 'ps/meson-build-perf-bench'
If you're seeing this message, it's not because your visit has been flagged as suspicious - but rather because the administrator of this website has taken proactive steps to protect its server from the scourge of AI-powered scrapers that have been aggressively targeting websites.
The system in place is called Anubis, a compromise solution designed to deter mass scraper activity without putting undue strain on legitimate users. Anubis uses a Proof-of-Work scheme similar to Hashcash, a proposed proof-of-work mechanism aimed at reducing email spam. In theory, this adds a negligible burden at an individual scale, but when scaled up, it becomes prohibitively expensive for malicious actors.
The ultimate goal of Anubis is not to stop scrapers entirely, but rather to buy time for the development of more sophisticated fingerprinting techniques that can identify legitimate headless browsers. By delaying the presentation of the challenge proof-of-work page, Anubis aims to present a 'good enough' placeholder solution, allowing researchers to focus on developing more effective solutions.
However, there's a catch. Anubis requires users to enable JavaScript, which is currently not possible with plugins like JShelter that disable modern JavaScript features. This presents an interesting paradox: AI companies have altered the social contract around website hosting, making it essential for legitimate users to enable JavaScript just to access certain websites.
The 'no-JS solution' is still in development and serves as a reminder that security measures are constantly evolving. As researchers work to develop more effective solutions, Anubis plays an important role in bridging the gap until then.