Show HN: SSH-hypervisor – like SSH, but each user gets their own microVM

Tackling a larger systems programming project with AI tools. This weekend I tried to make a hypervisor hooked up to SSH. It’s like: But every time someone logs in with a different name, instead of being a user on the host machine, it greets you and then spins up a virtual machine with Firecracker.

This isn’t an original idea, by the way! I had seen this somewhere online, with a person showing off their tiny OS with Firecracker microVMs over public SSH. Unfortunately I don’t remember where I saw this, but I wanted to take this idea and make it a bit whimsical, while adding a couple toy features.

Update: A er shared the project https://github.com/nuta/kerla Back in high school and college, I used to make a lot of smaller, fun projects over the weekend and share them with people. I don’t do this as much now with a job. These tiny projects became less interesting as I grew familiar with systems; more implementation-heavy rather than new ideas.

I think that’s sad though. This project would maybe have taken me 1-2 weeks in the past, so I was hoping that with AI tools, I could do it in just a weekend (inspiration). Then I can spend time on more frivolous projects. I still get ideas all the time.

This is one of them, let’s just build it, see where it goes and let my creative side take control!

“Hypervisor is essentially a hardware-assisted catch block”. This is all what I want you to learn from this book. Hardware-assisted hypervisors are event handlers. They are not like a CPU emulator.

In JavaScript, the life of a hypervisor looks like this: A hypervisor runs the guest OS in a try block, catches events (VM exits), and goes back to the guest mode again.

I want to keep this in mind while working through the project. Firecracker is a very lightweight hypervisor, and they spin up “microVMs” — since hypervisors are catch-blocks, that essentially means the catch-block is small. Firecracker only emulates a few devices and relies on host features for as much as possible.

This makes it really fast to boot compared to QEMU. However, this doesn’t mean that Firecracker is any simpler to set up than other hypervisors. You still need to hook up all the parts of a virtual computer in the right places to get things working!

For instance: Bring your own init system like OpenRC / Systemd. Attach a kernel ramfs, disk at startup. Want network? Set up a MAC address, TAP device, bridge, IP routing rules, firewall filters, packet forwarding, and so on.

Want multiple VMs? Create a network bridge, set the controller of the TAP to that bridge, allocate private IPs from a pool, dynamically configure iptables. Want serial logs? Edit your kernel boot arguments to send them at a baud rate over the /dev/console TTY.

It’s a good reminder that VMs are tiny little computers that live in your own. When you start up VMs, you’re building up your own computer from scratch!

I found out in this project that, while AI tools made coding a lot faster (thousands of lines in minutes), they didn’t speed up debugging on the systems side.

During this whole debugging session (5+ hours), I asked ChatGPT a lot of stuff. Gave up on Claude Code since it kept making changes. The AI very confidently guided me toward directions that didn’t work, and it gave me a lot of false hope.

But it did eventually find the issue, which was the lack of random entropy causing silent blocking, which I wouldn’t have found otherwise without Google search or strace. I think it probably saved time overall?

Then, I spent another hour trying to get this working with OpenRC. It does not work. I’m just going to call it quits and use bash as my init process, oh well.

And then! It’s working now, but SSH still takes 6 seconds to start up. virtio-rng and building my own vmlinux

Remember the entropy device from earlier? I still have this rngd hack in my init script that initializes fake entropy:

Lately it’s become clear that this is a bad idea, and it adds exactly 5 seconds to VM startup for starting a “jitter” generator, which makes the time between boot and getting a shell ~4x slower.

I work out how to add an entropy device by manually hitting the Firecracker HTTP endpoint, but it’s still not appearing as /dev/hwrng on the guest. I think this is because the guest kernel that I’m using is from the quickstart_guide public bucket in S3, and it’s a very old Linux 4.14 image without many devices.

Or maybe not. In any case, if I have a newer Linux version then random.trust_cpu (introduced in Linux 4.19) will be respected, and I shouldn’t have a problem either way since it can rely on hardware RNG instructions.

So I try to build an image based on Linux 6.1, and I run into—problems!

The “Cannot open root device” is a completely useless error message that could mean any number of things, whether APIC issues or uninitialized modules, or even Firecracker bugs.

The AI is equally confused.

I spend about an hour stuck on this for a while, trying different Linux and Firecracker versions and flipping kernel configs on/off.

At this point, it’s Monday. So I go to work. And while I’m on the train there, I do some Googling and find this bit from kernel-policy.md:

We use these configurations to build microVM-specific kernels vended by Amazon Linux … As a result, kernel configurations found in this repo should be used to build exclusively the aforementioned Amazon Linux kernels.

We do not guarantee that using these configurations to build upstream kernels, will work or produce usable kernel images.

Okay, so that’s it. I need to use Amazon Linux, and then it will work, right? Of course the people at Amazon would use their own Linux fork.

So it’s back to the AI. Let’s run this build again, using Orbstack for their seamless VMs on macOS. Now that we’re building from Amazon Linux, it should work with Firecracker, right?

And it fails again — but I then removed the pci=off acpi=off options, and this combined with Amazon Linux allows it to finally boot. Hooray.

Even better, I’m no longer on an ancient Linux version. Timidly, I decide to try returning to OpenRC despite my issues from earlier. And yes: OpenRC works, sshd is running, and even agetty is finally no longer stalling.

Everything is blissful. Yay! It all makes sense again, definitely worth debugging.

Now that things boot, everything just got a lot easier.

I also build the vmlinux kernel for ARM64, just for fun, again inside an Orbstack VM.

Hooking Firecracker up to an SSH server

We have VMs working! It’s time to hook it up to SSH and build our app.

I’m relying heavily on the AI to figure out the implementation on this, and it’s going swimmingly. It worked out the SSH protocol with no issues at all, and it’s especially good at making cute interactive terminal output, like animated progress bars.

With things like session management and architecture, it’s also good to work in broad strokes as we make changes. (At some point my VM ran out of RAM and started thrashing.)

But for the most part, this was pretty easy to code since nothing was too tricky to debug on the application side.

Just kept iterating, trying it out and fixing things that didn’t quite look right.

It works! And it is very cute :)

I am still hosting this at vmcity.ekzhang.com for now, but I will stop at some point, earlier if I notice any crypto miners or other unscrupulous folk.