Merge branch 'sj/string-list' - Protecting Against Bot Scraper Attacks
The administrator of this website has taken proactive measures to safeguard against the scourge of AI companies aggressively scraping websites, resulting in downtime for our server and making resources inaccessible to everyone.
To combat this issue, a cutting-edge security solution called Anubis has been set up on our server. This innovative system uses a Proof-of-Work scheme similar to Hashcash, which was initially proposed as a solution to reduce email spam. At the individual scale, the additional load is negligible, but at mass scraper levels, it becomes prohibitively expensive for attackers.
So, what exactly does Anubis do? In essence, it's a compromise that adds an extra layer of protection to our server. When you visit our website, you'll notice that you need to complete a simple challenge - usually displayed as a CAPTCHA-like puzzle or a numerical sequence. This is where the Proof-of-Work scheme comes into play.
The idea behind Anubis is to create a situation where individual users may not even notice the added load, but mass scraper attacks become much more difficult and expensive. Essentially, it's a "good enough" placeholder solution that allows our team to focus on more advanced methods of fingerprinting and identifying legitimate headless browsers (such as those used by social media platforms).
However, there are some caveats to keep in mind. Anubis requires the use of modern JavaScript features, which plugins like JShelter may disable. To overcome this challenge, you'll need to enable JavaScript on your device and refrain from using such plugins while visiting our website.
Unfortunately, a no-JS solution is still in development, so we must ensure that users have the necessary permissions to access JavaScript. This new standard for website hosting is an evolving aspect of the digital landscape, and we're committed to staying ahead of these emerging threats.
Thank you for your understanding and cooperation as we navigate this complex issue. We appreciate your patience and look forward to continuing to provide a safe and engaging experience for all our users.