Encorebuilds and deploys backend applications, and since mid-2022 every one of those builds has run inside aFirecrackermicroVM. Firecracker strips the emulated hardware down to what a Linux kernel needs, giving each build the isolation of a virtual machine with startup close to the cost of a container.
Firecracker drivesKVM, so it needs a Linux host with/dev/kvm, which no Mac has, and most engineers at Encore develop on a Mac. The maintainers have no plans to close that, given they turned down a working proof of concept built on Apple’s Virtualization.framework and said theydo not plan to support macOS any time soon.
So for four years, working on the build system meant working on it somewhere else. We wanted to run the same build system on our laptops while keeping Firecracker in production, so we builtcrackling, a single microVM API that drives Firecracker on Linux and Apple’s hypervisor on macOS; booting the same images on both required rebuilding much of the Linux image toolchain to run on macOS.
Images were the hard half, because Firecracker boots a block device and Docker produces layers. We could not find an existing tool that converted Docker layers into a block device Firecracker could boot, so we built the conversion ourselves, half on your laptop and half over SSH:
That last line pipes a shell function into a login shell on the far end and runs it there. The bash was later rewritten in Go, but the pipeline and the host did not change. We hadsquash_layersre-extract every layer in manifest order, delete the.wh..wh..opqwhiteout markers withfindbecausetarwill not apply them for us, write a hardcoded/etc/resolv.confsince the VM had no DNS otherwise, pull the image’s environment variables out of the Docker config withjq, and finally callmksquashfsto produce something Firecracker could boot. We keyed a cache on the Docker image id so the whole path could be skipped when it matched. It ran whenever you had changed anything in the image, which was most of the time if you were working on the guest side.
Restarting took a third SSH, to kill your container and start its replacement:
Firecracker is running inside a Docker container there, so we had to pass–privileged,/dev/kvmand/dev/net/tunthrough, because the process inside that container was going to create tap devices and boot virtual machines of its own.
Firecracker expects those tap devices to attach to a host bridge, and inside a Docker container there is no host bridge. So we built one, in a shell script that ran beforebuildsvc, the build service itself, came up:
The script builds a bridge nameddocker0inside the container, enslaves the container’s owneth0to it, then moves the IP address offeth0and onto the bridge. The tap devices then had something to attach to that looked enough like a real Docker host.
The setup worked, which is why it lasted from 2022 until we finally replaced it. What it cost was harder to justify at a company whose whole pitch is that backend development should be smooth: that you should be able to write your application, run it locally, and have the infrastructure follow from the code rather than from a pile of YAML you maintain by hand. Meanwhile the part of our product that turns a git push into a running application was the one thing we could not run on the machines we write software on.
The loop was long enough that you thought twice before trying something speculative, and what we wanted was to run the build system on a Mac, natively, booting the same images, on the machine we were already sitting in front of.
We looked at what already existed first, and running Linux in a VM on macOS has several working implementations: Apple’s owncontainerreached 1.0 this June,LimaandTarthave been doing it for years, and podman can too, throughlibkrun. You can even get/dev/kvminside a Linux VM on an M3 or later running macOS 15, which runs Firecracker unmodified.