Testing

Keystone provides multiple VM-based testing workflows for validating configurations before deploying to physical hardware. These range from fast config iteration with build-vm to full-stack deployment testing with libvirt VMs that include TPM emulation, Secure Boot, and disk encryption. This page covers all testing modes, ISO generation for installation media, and debugging techniques.

Testing modes overview

| Mode | Tool | Speed | What it tests | Encryption/TPM/SecureBoot | | ---------------- | --------------------------------------------- | ---------- | ----------------------------------------------- | ------------------------- | | Fast config | bin/build-vm | ~1-2 min | Terminal and desktop configuration | No | | Full stack | bin/virtual-machine + bin/test-deployment | ~10-15 min | Complete deployment including security features | Yes | | Module isolation | MicroVM | ~2-5 min | Individual NixOS module behavior | Configurable | | Flake validation | nix flake check | ~30 sec | Nix expression correctness | N/A |

Fast configuration testing with build-vm

The bin/build-vm script provides the fastest way to test desktop and terminal configurations using nixos-rebuild build-vm. It builds the configuration and automatically connects to the VM.

Usage

# Terminal development environment - auto-SSH into VM
./bin/build-vm terminal

# Hyprland desktop - open graphical console
./bin/build-vm desktop

# Clean old artifacts first
./bin/build-vm terminal --clean
./bin/build-vm desktop --clean

# Build only, do not connect
./bin/build-vm terminal --build-only

How it works

Terminal VM: Starts the VM in the background, waits for SSH to be ready, and automatically connects. Exit SSH with exit or Ctrl-D (the VM keeps running in the background). Reconnect with ssh -p 2222 testuser@localhost. Stop the VM with kill $(cat build-vm-terminal.pid).
Desktop VM: Opens a QEMU window with the Hyprland desktop. Stop with poweroff inside the VM or Ctrl-C.

Key characteristics

Mounts the host Nix store via 9P, resulting in faster builds with no copying overhead.
Creates a persistent disk (build-vm-{terminal,desktop}.qcow2) that survives reboots but can be deleted with --clean.
Uses QEMU directly without libvirt.
Simple credentials: user testuser/testpass, root root/root.
Terminal VM forwards SSH to localhost:2222.
No encryption, Secure Boot, or TPM overhead. Focus is on configuration correctness.

When to use build-vm

Use build-vm for testing desktop configurations, terminal development environment changes, home-manager modules, and any scenario where fast iteration matters more than testing security features. For full-stack testing including ZFS encryption, Secure Boot, and TPM, use the libvirt-based workflow described below.

Full-stack testing with libvirt VMs

The bin/virtual-machine script creates and manages libvirt VMs with UEFI Secure Boot, TPM 2.0 emulation, and full disk encryption support. This is the primary tool for validating the complete Keystone deployment pipeline.

Host requirements

The host system must have libvirtd enabled with the current user in the libvirtd group:

virtualisation.libvirtd.enable = true;
users.users.<youruser>.extraGroups = [ "libvirtd" ];

QEMU with UEFI/OVMF firmware support is required. At least 4 GB of RAM and 2 vCPUs are recommended for the VM.

Creating and managing VMs

# Create a new VM with default settings and start it
./bin/virtual-machine --name keystone-test-vm --start

# Create VM with custom resources
./bin/virtual-machine --name large-vm --memory 8192 --vcpus 4 --disk-size 50 --start

# Create VM with a specific ISO
./bin/virtual-machine --name my-vm --iso /path/to/custom.iso --start

# Post-installation: snapshot disk, remove ISO, and reboot
./bin/virtual-machine --post-install-reboot keystone-test-vm

# Reset Secure Boot back to Setup Mode
./bin/virtual-machine --reset-setup-mode keystone-test-vm

# Completely delete VM and all associated files
./bin/virtual-machine --reset keystone-test-vm

VM features

UEFI Secure Boot in Setup Mode -- VMs boot with Secure Boot firmware enabled but no Platform Key enrolled, allowing the Keystone installer to enroll custom keys. See Secure Boot for details on the enrollment process.
TPM 2.0 emulation -- Software TPM for testing TPM-based disk unlock.
Static IP -- 192.168.100.99 on the keystone-net libvirt network.
Serial console and SPICE graphics -- Multiple connection methods for different debugging scenarios.

Connecting to VMs

# Graphical display via SPICE
remote-viewer $(virsh domdisplay keystone-test-vm)

# Serial console (useful for boot debugging)
virsh console keystone-test-vm
# Press Ctrl+] to exit

# SSH (after NixOS installation) - use the helper script
./bin/test-vm-ssh

# SSH with a command
./bin/test-vm-ssh "systemctl status"

The bin/test-vm-ssh script connects to the VM at 192.168.100.99 with an isolated known_hosts file to avoid polluting the user's SSH configuration. It filters out SSH warnings and verifies the VM is running before attempting connection.

Verifying Secure Boot Setup Mode

Inside the VM, verify the Secure Boot state:

bootctl status

Expected output for Setup Mode:

System:
     Firmware: UEFI 2.70 (EDK II 1.00)
  Secure Boot: disabled (setup)
   Setup Mode: setup

After the installer enrolls keys, the output changes to:

System:
     Firmware: UEFI 2.70 (EDK II 1.00)
  Secure Boot: enabled (user)
   Setup Mode: user

VM configuration details

Configuration: .#test-server in flake.nix
Hostname: keystone-test-vm
Disk: /dev/vda (20 GB default)
NVRAM file: vms/<name>/OVMF_VARS.fd (540,672 bytes when fresh, indicating no pre-enrolled keys)
Graphics: QXL with SPICE for Hyprland compatibility in VMs (virtio-gpu requires GL acceleration, which is unavailable with NVIDIA hosts due to NixOS issue #164436)

Automated deployment testing

The bin/test-deployment script orchestrates a complete fresh deployment test:

Stops and resets any existing test VM.
Optionally rebuilds the ISO with SSH keys.
Creates and starts the VM from the ISO with TPM emulation.
Waits for boot and SSH access.
Deploys via nixos-anywhere to .#test-server.
Unlocks disk via initrd SSH.
Runs Secure Boot provisioning.
Verifies the deployment (SSH, ZFS, Secure Boot).

# Normal test run
./bin/test-deployment

# Rebuild ISO first
./bin/test-deployment --rebuild-iso

# Force VM cleanup before testing
./bin/test-deployment --hard-reset

# Show full output for debugging
./bin/test-deployment --debug

A full deployment test takes approximately 10-15 minutes.

Incremental updates

For iterative development, bin/update-test-vm pushes configuration changes to a running VM without redeploying:

# Quick update, no reboot
./bin/update-test-vm

# Update and reboot (for boot-related changes)
./bin/update-test-vm --reboot

# Build on the VM instead of locally
./bin/update-test-vm --build-host

This takes approximately 1-3 minutes and preserves all VM state including enrolled keys, data, and TPM enrollment.

MicroVM testing

For lightweight, reproducible testing of specific NixOS module configurations, Keystone uses microvm.nix. MicroVMs provide fast feedback loops and integrate with the Nix build system.

How it works

microvm.nix is added as an input in tests/flake.nix.
Dedicated MicroVM configurations are defined in tests/microvm/ (e.g., tpm-test.nix for TPM emulation).
Test runner scripts in bin/ manage the MicroVM lifecycle, including host-side services (such as swtpm for TPM emulation) and post-boot verification.

Running a MicroVM test

nix develop --command bin/test-microvm-tpm

The test script handles building the MicroVM runner, starting required host services, launching the MicroVM, performing checks inside the guest, and cleaning up.

ISO generation

The bin/build-iso script generates Keystone installer ISOs with optional SSH key injection for remote installation.

Building an ISO

# Build without SSH keys
./bin/build-iso

# Build with SSH key from file
./bin/build-iso --ssh-key ~/.ssh/id_ed25519.pub

# Build with SSH key string directly
./bin/build-iso --ssh-key "ssh-ed25519 AAAAC3Nz... user@host"

# Build directly with Nix (no SSH keys)
nix build .#iso

The ISO includes SSH with injected keys, DHCP networking, essential tools (git, vim, parted), ZFS support, and nixos-anywhere compatibility.

Writing to USB

lsblk
sudo dd if=result/iso/*.iso of=/dev/sdX bs=4M status=progress
sync

Using the ISO

Boot from USB or attach to a VM.
The system auto-configures SSH, DHCP, and essential tools.
Obtain the IP address: ip addr show.
Connect: ssh root@<ip-address>.
Deploy the Keystone configuration from a development machine:
```
nixos-anywhere --flake .#your-config root@<installer-ip>
```

Screenshot-based debugging

For libvirt VMs, bin/screenshot captures the graphical display as a PNG image for debugging boot issues where SSH is unavailable:

# Screenshot the default VM (keystone-test-vm)
./bin/screenshot

# Screenshot a specific domain
./bin/screenshot keystone-test-vm

# Screenshot with custom output path
./bin/screenshot keystone-test-vm debug.png

The script outputs the path to the PNG file (e.g., screenshots/vm-screenshot-20251222-010214.png). This is useful for diagnosing UEFI boot failures, Secure Boot issues, disk unlock prompts in initrd, and any state where SSH is not yet available.

Make targets

Keystone provides Make targets that wrap the testing scripts for convenience. Run make help to see all available targets. Key testing targets:

# Fast config testing (no encryption/TPM)
make build-vm-terminal          # Build and SSH into terminal dev VM
make build-vm-desktop           # Build and open desktop VM

# Libvirt VM management (full TPM + Secure Boot)
make vm-create                  # Create and start VM with TPM
make vm-start                   # Start existing VM
make vm-stop                    # Stop VM gracefully
make vm-reset                   # Delete VM and all artifacts
make vm-ssh                     # SSH into test VM
make vm-console                 # Serial console access
make vm-display                 # Open graphical display
make vm-post-install            # Post-install workflow
make vm-reset-secureboot        # Reset to Secure Boot setup mode

# Test suites
make test                       # Run all tests
make test-checks                # Fast flake validation
make test-module                # Module isolation tests
make test-integration           # Integration tests
make test-deploy                # Full stack deployment test
make test-desktop               # Hyprland desktop test
make test-hm                    # Home-manager module test
make test-template              # Validate flake template

# ISO
make build-iso                  # Build installer ISO
make build-iso-ssh              # Build ISO with SSH key

Use VM_NAME=my-vm make vm-ssh to target a VM with a non-default name.

Common testing workflows

Fresh deployment validation

./bin/test-deployment
./bin/test-vm-ssh
bootctl status              # verify Secure Boot
zpool status rpool          # verify ZFS pool
systemctl status            # verify services

Iterative module development

# Edit module
vim modules/os/users.nix

# Push change to running VM
./bin/update-test-vm

# SSH and verify
./bin/test-vm-ssh "systemctl status"

Boot-related changes

# Edit boot configuration
vim modules/os/storage.nix

# Update and reboot
./bin/update-test-vm --reboot

# Verify boot behavior
./bin/test-vm-ssh

Complete reset and redeploy

./bin/test-deployment --hard-reset --rebuild-iso

Troubleshooting

VM will not start

# Check VM status
virsh list --all | grep keystone-test-vm

# Verify libvirtd is running
systemctl status libvirtd

# Check the libvirt network
virsh net-list | grep keystone-net

# Verify OVMF firmware is available
ls -la /nix/store/*-OVMF-*/FV/

SSH connection fails

# Check VM IP
virsh domifaddr keystone-test-vm

# Verify network leases
virsh net-dhcp-leases keystone-net

# Fall back to serial console
virsh console keystone-test-vm

Deployment hangs at disk unlock

If the deployment appears stuck, the VM may be waiting for a disk unlock passphrase. Connect via serial console:

virsh console keystone-test-vm

Enter the passphrase (default for test VMs: keystone).

Hyprland does not start in VM

Verify qemu-guest.nix is imported in the VM configuration. Without it, virtio drivers and graphics backends may be missing.
Check kernel modules are loaded: lsmod | grep virtio.
Review compositor logs: journalctl --user -u hyprland.
Review system logs for graphics issues: journalctl -b | grep -i "drm\|egl\|opengl".

Update fails

Roll back on the VM:

./bin/test-vm-ssh "sudo nixos-rebuild switch --rollback"

Or reset and redeploy:

./bin/virtual-machine --reset keystone-test-vm
./bin/test-deployment

Testing checklist

Before committing changes:

Run ./bin/test-deployment to verify fresh deployment works.
Test feature-specific workflows (e.g., TPM enrollment, user creation).
Verify documentation matches actual behavior.
Test error handling (disable Secure Boot, remove TPM, etc.).
Test rollback: ./bin/test-vm-ssh "nixos-rebuild switch --rollback".

After major changes:

Test with --rebuild-iso to ensure ISO generation works.
Verify Secure Boot enrollment completes.
Test initrd SSH unlock.
Confirm automatic boot unlock if TPM is enrolled.

Testing

Testing modes overview

Fast configuration testing with build-vm

Usage

How it works

Key characteristics

When to use build-vm

Full-stack testing with libvirt VMs

Host requirements

Creating and managing VMs

VM features

Connecting to VMs

Verifying Secure Boot Setup Mode

VM configuration details

Automated deployment testing

Incremental updates

MicroVM testing

How it works

Running a MicroVM test

ISO generation

Building an ISO

Writing to USB

Using the ISO

Screenshot-based debugging

Make targets

Common testing workflows

Fresh deployment validation

Iterative module development

Boot-related changes

Complete reset and redeploy

Troubleshooting

VM will not start

SSH connection fails

Deployment hangs at disk unlock

Hyprland does not start in VM

Update fails

Testing checklist

See also