> Blog Post

Running a Fleet of Firecracker microVMs for eve.dev Agents, Part 3: Packaging an Agent as a microVM Image

awsaws-cdkfirecrackereveagentstypescripts3ai-infrastructure

At the end of Part 2 we had a fleet of bare-metal hosts, each running Firecracker and a host-agent, each announcing its free capacity into a registry — and not a single agent running on any of them. That is the gap this part closes. A host cannot boot "an eve agent"; Firecracker boots a kernel and a root filesystem. So before the control plane can place anything, we need to turn an agent directory into a bootable image and put it somewhere every host can reach.

By the end of Part 3 you will have a build-rootfs.sh that packages any eve agent into a versioned ext4 image, a CDK-provisioned S3 bucket that stores those images (plus the shared kernel), and — the part that makes per-session microVMs actually fast — a Firecracker snapshot flow that resumes a warm agent in well under a second.

What a microVM Image Actually Is

In the single-VM guide we booted a stock Ubuntu rootfs and then, after it was running, installed Node and ran npx eve by hand inside the guest. That is fine when there is one microVM and a human at the console. It falls apart on a fleet: a host booting a microVM has no human to run install commands, and re-installing Node and npm-installing an agent on every single boot would make each launch take minutes and depend on the network.

So we invert it. We do the slow work — install the runtime, install the agent, wire up the init — once, ahead of time, and bake the result into the rootfs. A fleet image is a filesystem that boots straight into a running agent with nothing left to install. Concretely, a bootable microVM is two files:

  • A kernel (vmlinux.bin) — the same uncompressed Linux kernel for every agent. We store one copy in the bucket and reuse it everywhere.
  • A rootfs (rootfs.ext4) — an ext4 filesystem with a minimal userland, Node, the agent's code, and a systemd unit that starts the agent on boot. This is the per-agent, per-version artifact.

Everything below builds and stores those two things.

The Artifact Bucket

First the place they live. It is an ordinary S3 bucket, but three settings matter: it is versioned (so a bad image is one aws s3 cp to roll back, not a rebuild), it is fully private (images are yours; nothing here is ever public), and it has a lifecycle rule so old noncurrent versions do not accumulate forever. This is the bucket the host IAM role in Part 2 already granted read access to — the name matches config.fleetName so that forward reference resolves.

// lib/artifact-stack.ts
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as s3 from "aws-cdk-lib/aws-s3";
import { config } from "./config";

export class ArtifactStack extends cdk.Stack {
  public readonly bucket: s3.Bucket;

  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    this.bucket = new s3.Bucket(this, "AgentImages", {
      bucketName: `${config.fleetName}-agent-images`,
      versioned: true,
      blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
      encryption: s3.BucketEncryption.S3_MANAGED,
      enforceSSL: true,
      lifecycleRules: [
        // Keep rollback headroom without hoarding every old image forever.
        { noncurrentVersionExpiration: cdk.Duration.days(30) },
      ],
      // Tutorial convenience — use RETAIN (and drop autoDeleteObjects) in prod.
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      autoDeleteObjects: true,
    });

    new cdk.CfnOutput(this, "BucketName", { value: this.bucket.bucketName });
  }
}

The layout inside the bucket is a convention the build script and the host-agent both agree on:

eve-fleet-agent-images/
  kernel/vmlinux.bin                         # shared by every agent
  host-agent/current/host-agent              # the daemon from Part 2
  agents/<name>/<version>/rootfs.ext4        # immutable, version = git SHA
  agents/<name>/<version>/snapshot/{snap,mem} # optional warm snapshot
  agents/<name>/current/rootfs.ext4          # movable "latest" pointer

The shared kernel goes in once — grab the same one the single-VM guide used and upload it:

ARCH=$(uname -m)
curl -fsSL -o vmlinux.bin \
  "https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/${ARCH}/kernels/vmlinux.bin"
aws s3 cp vmlinux.bin s3://eve-fleet-agent-images/kernel/vmlinux.bin

build-rootfs.sh

Here is the script that turns an agent directory into a bootable rootfs. It creates an empty ext4 filesystem, lays a minimal Ubuntu userland into it with debootstrap, installs Node and the agent, drops in the systemd unit that runs the agent on boot, and uploads the result to the versioned bucket. Run it on any Linux machine with root — a CI runner, or one of the fleet hosts.

#!/usr/bin/env bash
# build-rootfs.sh <agent-dir> <version>
#   e.g. ./build-rootfs.sh ./my-agent "$(git rev-parse --short HEAD)"
set -euo pipefail

AGENT_DIR="$1"
VERSION="$2"
AGENT_NAME="$(basename "$AGENT_DIR")"
BUCKET="eve-fleet-agent-images"
ROOTFS="rootfs.ext4"
SIZE_MB=2048

# 1. An empty ext4 filesystem, sized for the userland + Node + the agent.
dd if=/dev/zero of="$ROOTFS" bs=1M count="$SIZE_MB"
mkfs.ext4 -F "$ROOTFS"

# 2. Mount it and lay down a minimal Ubuntu base.
mnt="$(mktemp -d)"
sudo mount -o loop "$ROOTFS" "$mnt"
sudo debootstrap --arch amd64 --variant=minbase jammy "$mnt" \
  http://archive.ubuntu.com/ubuntu/

# 3. Install the Node runtime and the agent itself, baked in for good.
sudo chroot "$mnt" /bin/bash -eux <<'CHROOT'
  apt-get update
  apt-get install -y curl ca-certificates
  curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
  apt-get install -y nodejs
  useradd -m -s /usr/sbin/nologin agent
CHROOT
sudo cp -r "$AGENT_DIR" "$mnt/opt/agent"
sudo chroot "$mnt" npm --prefix /opt/agent install --omit=dev

# 4. The init: start the agent on boot. Secrets are NOT baked in — they are
#    fetched per-microVM from MMDS at boot (see the next section).
sudo tee "$mnt/etc/systemd/system/eve-agent.service" >/dev/null <<'UNIT'
[Unit]
Description=eve agent
After=network-online.target
[Service]
Type=simple
User=agent
WorkingDirectory=/opt/agent
ExecStartPre=/usr/local/bin/fetch-mmds-env /run/agent.env
EnvironmentFile=/run/agent.env
ExecStart=/usr/bin/npx eve start --host 0.0.0.0 --port 8080
Restart=always
[Install]
WantedBy=multi-user.target
UNIT
sudo chroot "$mnt" systemctl enable eve-agent.service

# 5. Unmount and publish: the immutable versioned copy, then move "current".
sudo umount "$mnt"
aws s3 cp "$ROOTFS" "s3://$BUCKET/agents/$AGENT_NAME/$VERSION/rootfs.ext4"
aws s3 cp "$ROOTFS" "s3://$BUCKET/agents/$AGENT_NAME/current/rootfs.ext4"
echo "published agents/$AGENT_NAME/$VERSION/rootfs.ext4"

A few decisions worth calling out. The agent runs as an unprivileged agent user, not root, inside the guest — a second layer of containment under the microVM boundary itself. It listens on 0.0.0.0:8080, which is the port the host-agent proxies out on the 8000–8999 range the Part 1 security group opened for ALB traffic. And the version is passed in — we use a git short SHA — so agents/my-agent/<sha>/rootfs.ext4 is immutable and content-addressed, while current/ is just a movable pointer.

Secrets: Injected Per-microVM, Never Baked In

An agent needs credentials — a model API key at minimum. Those must never live in the rootfs: the image is shared across every boot of that agent, it sits in S3, and baking a key into it means rotating the key means rebuilding the image. Firecracker gives us the right tool here — the microVM Metadata Service (MMDS), a small metadata store the host populates per-microVM and the guest reads over a link-local address, exactly like EC2's IMDS.

The flow: when the control plane (Part 4) tells a host to boot an agent, the host-agent writes that agent's secrets into the microVM's MMDS before starting it. Inside the guest, the fetch-mmds-env helper in the systemd unit reads them at boot into /run/agent.env (a tmpfs file that never touches disk):

#!/usr/bin/env bash
# /usr/local/bin/fetch-mmds-env <out-file> — baked into the image
set -euo pipefail
token=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \
  -H "X-metadata-token-ttl-seconds: 60")
curl -s -H "X-metadata-token: $token" \
  "http://169.254.169.254/agent/env" > "$1"

The image stays generic and cache-friendly; the secrets are per-instance and rotate without a rebuild. This is the clean seam between "what the agent is" (the image) and "what this run of it is allowed to do" (the MMDS payload).

How a Host Boots One of These

This is the point where Part 3 meets the single-VM guide. Given the built rootfs and the shared kernel — both pulled from S3 into a local cache — the host-agent writes the same vm-config.json shape from the single-VM guide, now pointing at the fleet artifacts:

{
  "boot-source": {
    "kernel_image_path": "/var/cache/eve/kernel/vmlinux.bin",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "/var/cache/eve/agents/my-agent/current/rootfs.ext4",
      "is_root_device": true,
      "is_read_only": false
    }
  ],
  "machine-config": { "vcpu_count": 2, "mem_size_mib": 1024 },
  "mmds-config": { "version": "V2", "network_interfaces": ["eth0"] }
}

The only real addition over the single-VM config is the mmds-config block that enables the metadata service the secrets flow depends on. The vcpu_count / mem_size_mib here are exactly the numbers Part 2's capacity model bins against — the image and the scheduler agree on an agent's size because it is written in one place.

Snapshots: From Cold Boot to Warm Resume

A cold boot of this image still does real work: the kernel initializes, systemd comes up, Node starts, and the eve runtime loads. That is a few seconds — fine for a long-lived agent, painful if you boot a fresh microVM per session. Firecracker's answer is snapshots: boot the image once, let it warm up, pause it, and capture its entire state — memory and device state — to files. A host can then resume from that snapshot instead of booting, and resume is sub-second because there is nothing to initialize; the agent is already running and listening.

We build the snapshot once, at image-build time, on a builder host:

# Boot the freshly built image, let the agent finish starting, then pause it.
curl --unix-socket /tmp/fc.sock -X PATCH "http://localhost/vm" \
  -d '{"state": "Paused"}'

# Capture memory + device state to two files.
curl --unix-socket /tmp/fc.sock -X PUT "http://localhost/snapshot/create" \
  -d '{
        "snapshot_type": "Full",
        "snapshot_path": "snap.file",
        "mem_file_path": "mem.file"
      }'

# Store the warm snapshot alongside the rootfs it was taken from.
aws s3 cp snap.file "s3://$BUCKET/agents/$AGENT_NAME/$VERSION/snapshot/snap"
aws s3 cp mem.file  "s3://$BUCKET/agents/$AGENT_NAME/$VERSION/snapshot/mem"

A host with the snapshot cached restores it with a single PUT /snapshot/load instead of the boot-source config above, and the agent is answering requests almost immediately. There are real tradeoffs to know: a snapshot is bound to the exact kernel and machine config it was taken with, so it is per-agent-version, not universal; and the memory file is the size of the microVM's RAM, so snapshots cost storage. But for the "boot a microVM per session" pattern the deploy workflow will want in Part 6, warm resume is the difference between a usable platform and a laggy one. If you skip snapshots, everything still works — hosts just cold-boot the rootfs.

Versioning and Rollback

Because every build writes an immutable agents/<name>/<version>/ path and only moves current/, the operational story is boring in the best way:

  • Deploy a new version: build-rootfs.sh ./my-agent <new-sha> — publishes the versioned copy and advances current.
  • Roll back: copy an older version's objects over current/ — no rebuild, seconds to take effect.
  • Audit: the versioned bucket keeps the history; current always names exactly what hosts will boot next.

The control plane in Part 4 records which version each placement should run, so "roll back agent X to version Y" becomes a control-plane operation, not an S3 scramble — but it all rests on this immutable-version-plus-pointer layout.

Build One and Look

With the artifact stack deployed and the kernel uploaded, package the sample agent from the single-VM guide:

cdk deploy EveFleetArtifacts
./build-rootfs.sh ./my-agent "$(git rev-parse --short HEAD)"

aws s3 ls --recursive s3://eve-fleet-agent-images/agents/my-agent/

You should see the versioned rootfs.ext4 and the current/ pointer. That is a bootable agent image sitting where every host can reach it — which is exactly what the control plane needs to exist before it can place one.

Tearing Down

The artifact bucket is versioned, so it will not delete while it holds object versions. In this stack autoDeleteObjects: true handles that for you on cdk destroy EveFleetArtifacts; in a real setup you would empty it deliberately. As always, mind the order — nothing that reads the bucket (the hosts) should outlive an attempt to remove it.

What's Next

You now have the missing half of a runnable agent: an image pipeline that bakes an eve agent into a versioned, bootable rootfs, a private bucket that stores it next to a shared kernel and optional warm snapshots, a per-microVM secrets path through MMDS, and a rollback story that is a pointer move. The hosts from Part 2 can pull any of it.

In Part 4 we build the brain that ties the two together: the control plane. API Gateway and Lambda over a DynamoDB registry that knows which hosts have capacity (Part 2) and which image versions exist (Part 3), a placement algorithm that bin-packs a new agent onto a host with room, and the command that tells that host's host-agent to boot the microVM. That is the part where deploy my-agent finally produces a running agent.

If you have not built a single microVM by hand yet, Running an eve agent in a Firecracker microVM on AWS is where the rootfs-and-kernel mechanics this part automates are walked through one step at a time.