Java Virtual Threads Ate My Memory: A Web Crawler's Tale of Speed vs. Memory

I've always been fascinated by Java's new features, and Virtual Threads are one of my favorites. But when I decided to build a simple web crawler using Virtual Threads, things didn't quite go as planned. In this article, I'll share my experience with using Virtual Threads for concurrency and how it can lead to memory issues if not managed properly.

From Platform Threads to Virtual Threads

I started by creating a basic web crawler using traditional platform threads. The task consisted of fetching URLs from a local server, processing the content, and moving on to the next one. I submitted a list of 20K URLs to be processed by 200 platform threads, which was a relatively simple test scenario.

But then, curiosity struck: "What happens if I use Virtual Threads instead?"

The Results Were Surprising

I ran the same logic using Virtual Threads, and at first, it seemed like a dream come true. The crawler was processing URLs at an incredible rate, what felt like milliseconds per page!

As you can see from the VisualVM stats, the crawler was indeed processing URLs much faster than with platform threads.

The OutOfMemoryError: A Lesson in Resource Management

However, my excitement was short-lived. The JVM gave up with an OutOfMemoryError! It turned out that Virtual Threads required significantly more memory than I had anticipated.

This experience taught me a valuable lesson about the importance of resource management when using Virtual Threads. Traditional patterns and assumptions from platform threads no longer apply, and we need to be more explicit about managing concurrency limits and resource constraints.

How to Avoid Memory Issues with Virtual Threads

So, how can we fix this without giving up on Virtual Threads? We can introduce a Semaphore to limit the number of concurrent tasks in flight. Before launching a new task, acquire a permit. Release it after processing completes. If no permits are available, the Virtual Thread is blocked.

Additional Tips for Avoiding Memory Issues

We should also avoid submitting too many tasks at the same time. In our test scenario, we submitted all 10,000 URLs at once, which was an artificial burst that rarely happens in production. In realistic applications, work arrives continuously over time. Implementing rate limiting or spreading the arrival of scraping requests over time might prevent overwhelming the crawler.

Conclusion

Virtual Threads are incredibly powerful, but they require us to be more explicit about resource management. The traditional patterns and assumptions that worked with platform threads may not apply. By understanding how Virtual Threads work and taking steps to manage concurrency limits and resource constraints, we can unlock their full potential while avoiding memory issues.