Merge branch 'jc/strbuf-split': Protecting Against AI Scrapers
As you're reading this article, it's likely that you've stumbled upon our site because of Anubis – the web server protection system designed to safeguard us against the relentless onslaught of AI-powered bots. The administrator has taken a proactive stance to ensure that our website remains secure and accessible to all users.
Anubis uses a Proof-of-Work scheme, inspired by Hashcash, a proposed solution for reducing email spam. This innovative approach aims to deter mass scraper attempts by making the process more expensive and time-consuming. At an individual scale, the additional load may seem insignificant, but when faced with large-scale scraping operations, it becomes a significant barrier. The ultimate goal is to create a "good enough" placeholder solution that buys time for further research and development of fingerprinting techniques, allowing us to better identify legitimate users without displaying the challenge proof-of-work page.
However, Anubis requires the use of modern JavaScript features, which may conflict with plugins like JShelter. To overcome this hurdle, please disable these plugins on our domain. We currently run Anubis version 1.21.3, and as a result, users must enable their JavaScript settings to proceed.
The changing landscape of website hosting has forced us to adopt more stringent security measures. Unfortunately, finding an AI-neutral solution is still in the works. For now, we must rely on JavaScript-based solutions to protect our site against these malicious actors.