Building a VMM from scratch, #0.5: Booting Linux in 190 lines of C

In #0 we set out to replace libkrun with a VMM of our own. Its boot path is landing one PR at a time, and its tests already boot a real Linux kernel to an init on four vCPUs (#2052). Before the parts that walk through that Rust code, here is the whole path in miniature: vmmlite, 190 lines of C, not counting comments, that boot a stock Ubuntu kernel on KVM to a PID 1 that prints one line.

$ ./vmmlite bzImage initramfs.cpio
[    0.000000] Linux version 7.0.0-1014-gcp (buildd@lcy02-amd64-091) ...
...
[    1.757880] Run /init as init process
Hello from PID 1!
[    1.766049] reboot: Restarting system

About three seconds after it starts, vmmlite exits with status 0. It's C on purpose: one file, nothing between you and the ioctl() calls. Our Rust VMM follows the same path, with more care.

For learning only. vmmlite is not for production. It doesn't validate its input files, runs without a sandbox, and emulates only a serial port and a reset. To run code in a micro-VM, use a production VMM such as BoxLite.

1. Eight steps

main() is eight numbered sections. Four talk to KVM; the other four only write bytes into guest memory.

Figure 1. The eight sections of vmmlite.c. Sections 1, 2, 7 and 8 call KVM; sections 3 to 6 only write bytes into guest RAM.

KVM does the heavy lifting. It runs guest instructions on the real CPU and, after KVM_CREATE_IRQCHIP and KVM_CREATE_PIT2, emulates the interrupt controllers and the timer inside the host kernel. What's left for vmmlite is the job of a PC's firmware and bootloader, plus two devices: a serial port and a reset line.

2. The VMM is the bootloader

Nothing runs in the guest before Linux: no BIOS, no GRUB. Linux's 64-bit boot protocol allows that, if the loader leaves the machine as they would have:

A bzImage starts with a boot sector and 16-bit setup code, written for a BIOS. vmmlite checks the header and skips both:

if (memcmp(k + 0x202, "HdrS", 4) || !(k[0x236] & 1))   // the header's magic; XLF_KERNEL_64
  die("vmmlite: need a bzImage with a 64-bit entry\n");
size_t setup = ((k[0x1f1] ? k[0x1f1] : 4) + 1) * 512;  // setup_sects (0 means 4) + boot sector
memcpy(ram + KERNEL, k + setup, ksize - setup);

Figure 2 shows guest RAM when the vCPU starts. The addresses are the ones most VMMs we surveyed share, and the ones in BoxLite's VMM design.

Figure 2. Guest RAM at the first instruction, not to scale. The arrows show how the CPU and the kernel find each piece: registers, zero page fields, and a scan for the MP table's signature.

Long mode is a handful of control-register bits, set through KVM:

s.cr3 = PML4;                                          // the page tables
s.cr4 |= 1 << 5;                                       // CR4.PAE
s.cr0 |= 1u << 31 | 1;                                 // CR0.PG, CR0.PE: paging on
s.efer |= 1 << 10 | 1 << 8;                            // EFER.LMA, EFER.LME: long mode
IOCTL(cpu, KVM_SET_SREGS, &s);                         // apply
struct kvm_regs r = {
    .rip = KERNEL + 0x200,                             // the 64-bit entry point
    .rsi = ZERO_PAGE,                                  // boot protocol: RSI = the zero page
    .rflags = 2};                                      // interrupts off (bit 1 is always set)

At 0x100200 sits startup_64, the decompressor's 64-bit entry (head_64.S). It unpacks the real kernel and jumps to it. The vCPU exits along the way, for CPUID or the first touch of a page, but KVM handles those exits itself.

CPUID comes from KVM_GET_SUPPORTED_CPUID, with two bits set in leaf 1, as Firecracker does: bit 31, "running under a hypervisor", and bit 24, the TSC-deadline timer that KVM emulates.

3. Telling Linux what the machine has

Firmware also tells the kernel what it runs on: the CPUs, the interrupt controllers, and how interrupts are wired. Modern firmware uses ACPI tables. vmmlite writes the older, much smaller Intel MultiProcessor (MP) table: 240 bytes that list one CPU, an ISA bus, an IOAPIC at 0xFEC00000, and 16 routes from ISA IRQ n to IOAPIC pin n.

Linux looks for the table's _MP_ signature in a few fixed places, one of them the last KiB below 640 KiB (mpparse.c). vmmlite writes it at 0x9FC00, as Firecracker, libkrun and our Rust VMM (#2052) do, and leaves that range out of the E820 map. Without the table or ACPI, Linux turns the IOAPIC and SMP off, as the design doc explains.

The routes match how KVM wires its in-kernel chips by default: line n drives PIC input n and IOAPIC pin n (irq_comm.c). So IRQ 4 from the serial port lands where Linux expects it.

4. One device: the serial port

Then vmmlite calls KVM_RUN in a loop. It returns only when the guest does something KVM leaves to user space, and vmmlite answers three kinds of port I/O:

} else if (port >= 0x3f8 && port <= 0x3ff) {         // COM1; an IN reads what com1()
  *io = com1(vm, port & 7, run->io.direction, *io);  //   returns, an OUT ignores it
} else if (run->io.direction == KVM_EXIT_IO_IN) {    // the guest reads another port:
  memset(io, 0, run->io.size);                       //   it reads 0
} else if (port == 0x64 && *io == 0xfe) {            // keyboard controller reset:
  return 0;                                          //   the guest rebooted; done
}                                                    // any other port write: ignored

COM1 is a 16550 UART at ports 0x3F8 to 0x3FF. com1() emulates just enough of five registers for Linux's 8250 driver to find it (8250_port.c) and call it an original 8250.

The kernel's own messages are the easy part. The console code polls the line status register before each byte (8250_port.c), and com1() always answers 0x60, "transmitter empty".

PID 1 is different. Its write() lands in a tty buffer, and the driver drains that buffer from its interrupt handler: it turns on the "THR empty" interrupt (8250_port.c) and writes the next byte each time the UART reports it (8250_port.c). So com1() keeps one bit of state, a pending "THR empty" event, and drives IRQ 4 from it:

struct kvm_irq_level irq = {.irq = 4, .level = ier & 2 && thre};
IOCTL(vm, KVM_IRQ_LINE, &irq);                         // IRQ 4, up or down

Figure 3. How "Hello from PID 1!" leaves the VM. Every port access is a VM exit that vmmlite answers; IRQ 4 tells the driver to send more.

Each character costs at least three exits: IIR, LSR, then the byte. That's the price of emulating a device one register at a time, and why real VMMs move bulk I/O to virtio.

The last exit is the reset. init.c is three calls:

int main(void) {
  if (write(1, "Hello from PID 1!\n", 18) < 0) return 1; // fd 1: the console, ttyS0
  tcdrain(1);                                            // wait until the UART has sent it all
  reboot(RB_AUTOBOOT);                                   // restart: Linux pulses the i8042
                                                         //   reset line, and vmmlite exits
}

Linux tries an ACPI reset, finds no ACPI, and falls back to the keyboard controller: it writes 0xFE to port 0x64, which pulses the CPU's reset line (reboot.c). vmmlite takes that write as the end of the VM and returns 0.

5. Every line has a cause

Here's the boot on a Google Cloud VM with nested KVM and Ubuntu's 7.0.0-1014-gcp kernel, trimmed to the lines vmmlite caused:

The keyboard controller line is the one vmmlite pays for. Every read of its ports returns 0, so Linux waits for a byte that never comes (i8042.c): 0.73 seconds between i8042: Probing ports directly. and that error, out of 1.76 seconds to Run /init.

6. Run it yourself

You need x86_64 Linux with KVM and access to /dev/kvm, plus gcc with static glibc and cpio. vmmlite boots your host's own kernel:

git clone https://github.com/boxlite-ai/vmmlite && cd vmmlite
gcc -O2 -Wall -Wextra -o vmmlite vmmlite.c
gcc -O2 -Wall -Wextra -static -o init init.c
echo init | cpio -o -H newc > initramfs.cpio
sudo install -m 644 /boot/vmlinuz-$(uname -r) bzImage
./vmmlite bzImage initramfs.cpio

We've booted it on Intel with nested KVM on Google Cloud, and on AMD-V emulated by QEMU. The repo's CI boots a GitHub Actions runner's own kernel on every push.

7. What vmmlite leaves out

vmmlite is a map, not a VMM you should run. What it skips is what the rest of this series is about:

8. Come build it with us

• Read it in one sitting. vmmlite.c is 328 lines with its comments, and each section opens with a diagram or a table. If a comment is wrong, open an issue.
• Then read the real thing. The VMM design doc follows the same boot path, and #1698 lists the open M1 work.
• Follow along on the RSS feed, or find us on Discord.