> Blog Post

Running a Fleet of Firecracker microVMs for eve.dev Agents, Part 2: The Host Fleet

awsaws-cdkfirecrackereveagentstypescriptec2autoscalingai-infrastructure

In Part 1 we built the network the fleet lives in: a VPC, private subnets for the hosts, public subnets for the ALB and NAT, and the three security groups that encode who may talk to whom. It is a well-shaped empty room. This part puts machines in it.

By the end of Part 2 you will have an Auto Scaling Group of bare-metal hosts, each one booting with Firecracker installed and a host-agent daemon running — the small process that will (in later parts) boot and tear down microVMs on command and report how much capacity the host has left. This is where the fleet stops being an empty network and becomes a pool of machines waiting for agents.

What a Host Is Responsible For

Before the CDK, it is worth being precise about what runs on one of these boxes, because it drives every decision below. A host is a bare-metal EC2 instance that runs three things:

  • Firecracker — the binary that boots microVMs, exactly as in the single-VM guide. One firecracker process per running agent.
  • The host-agent — a daemon we run as a systemd service. It is the host's local control loop: it registers the host with the control plane on boot, heartbeats its free capacity, and accepts "boot agent X" / "tear down agent Y" commands on port 7000 (the management port we opened in Part 1's host security group). We build the control plane it talks to in Part 4; in this part we install the daemon and wire up registration so the host announces itself the moment it is healthy.
  • The microVMs themselves — one per agent, each its own Firecracker guest. Nothing runs them yet; Part 3 gives us the images and Part 4 the scheduler that places them.

The host is cattle. It has no durable state that matters — if it dies, the control plane reschedules its agents onto other hosts (Part 7). That is what lets us put the whole thing behind an Auto Scaling Group.

The IAM Role: Least Privilege for a Host

Each host assumes one IAM role, and it should be able to do exactly four things and nothing more: be reached over SSM (no SSH, same as Part 1's security group), publish metrics and logs to CloudWatch, read agent images from the artifact bucket (built in Part 3), and register/heartbeat itself in the host registry table (built in Part 4). Forward-referencing those two resources by name is fine — we control the naming through config.fleetName, so the ARNs are predictable before the resources exist.

// lib/host-fleet-stack.ts
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as iam from "aws-cdk-lib/aws-iam";
import * as autoscaling from "aws-cdk-lib/aws-autoscaling";
import { config } from "./config";

export interface HostFleetStackProps extends cdk.StackProps {
  /** Consumed from the network stack in Part 1. */
  vpc: ec2.IVpc;
  hostSg: ec2.ISecurityGroup;
}

export class HostFleetStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: HostFleetStackProps) {
    super(scope, id, props);
    const { vpc, hostSg } = props;

    const role = new iam.Role(this, "HostRole", {
      assumedBy: new iam.ServicePrincipal("ec2.amazonaws.com"),
      managedPolicies: [
        // SSM Session Manager — how humans and automation reach the host.
        iam.ManagedPolicy.fromAwsManagedPolicyName("AmazonSSMManagedInstanceCore"),
        // Publish host metrics + logs (the CloudWatch agent from the single-VM guide).
        iam.ManagedPolicy.fromAwsManagedPolicyName("CloudWatchAgentServerPolicy"),
      ],
    });

    // Read-only pull of agent rootfs images from the artifact bucket (Part 3).
    role.addToPolicy(
      new iam.PolicyStatement({
        actions: ["s3:GetObject", "s3:ListBucket"],
        resources: [
          `arn:aws:s3:::${config.fleetName}-agent-images`,
          `arn:aws:s3:::${config.fleetName}-agent-images/*`,
        ],
      })
    );

    // Register on boot + heartbeat capacity into the host registry (Part 4).
    role.addToPolicy(
      new iam.PolicyStatement({
        actions: ["dynamodb:PutItem", "dynamodb:UpdateItem", "dynamodb:GetItem"],
        resources: [
          `arn:aws:dynamodb:${this.region}:${this.account}:table/${config.fleetName}-hosts`,
        ],
      })
    );

    // ...launch template + ASG below
  }
}

Note what is not here: no ec2:*, no ability to touch other hosts, no write access to the image bucket. A compromised agent that somehow escaped its microVM would find a host role that can read images and update its own registry row — nothing else.

The Launch Template and User Data

The launch template is the recipe every host in the ASG is stamped from. It pins the bare-metal instance type (the /dev/kvm requirement from the single-VM guide has not gone away — m5.metal is still the smallest thing that gives us KVM), attaches the host security group from Part 1, and runs user data that installs Firecracker and the host-agent.

We give it a roomy 200 GiB gp3 root volume: unlike the single-VM guide, a fleet host caches many agent rootfs images locally so repeat boots do not re-pull from S3, and images add up.

    const userData = ec2.UserData.forLinux();
    userData.addCommands(
      "set -euo pipefail",
      // --- Firecracker, exactly as in the single-VM guide ---
      "dnf install -y amazon-cloudwatch-agent",
      "usermod -aG kvm ec2-user",
      "ARCH=$(uname -m)",
      "rel=$(curl -s https://api.github.com/repos/firecracker-microvm/firecracker/releases/latest | grep browser_download_url | grep ${ARCH} | cut -d '\"' -f 4)",
      "curl -L ${rel} -o /tmp/firecracker.tgz",
      "tar -xzf /tmp/firecracker.tgz -C /tmp",
      "mv /tmp/release-*/firecracker-*-${ARCH} /usr/local/bin/firecracker",
      "chmod +x /usr/local/bin/firecracker",
      // --- The host-agent: your daemon, shipped as a release artifact ---
      `aws s3 cp s3://${config.fleetName}-agent-images/host-agent/current/host-agent /usr/local/bin/host-agent`,
      "chmod +x /usr/local/bin/host-agent",
      // It reads its config from the instance environment; the control-plane URL
      // is published as an SSM parameter by Part 4 and fetched here.
      "cat > /etc/systemd/system/eve-host-agent.service <<'EOF'",
      "[Unit]",
      "Description=eve fleet host-agent",
      "After=network-online.target",
      "[Service]",
      `Environment=FLEET_NAME=${config.fleetName}`,
      `Environment=HOST_RESERVED_VCPU=${config.hostReservedVcpu}`,
      `Environment=HOST_RESERVED_MEM_MIB=${config.hostReservedMemMib}`,
      "ExecStart=/usr/local/bin/host-agent",
      "Restart=always",
      "[Install]",
      "WantedBy=multi-user.target",
      "EOF",
      "systemctl daemon-reload",
      "systemctl enable --now eve-host-agent",
    );

    const launchTemplate = new ec2.LaunchTemplate(this, "HostTemplate", {
      instanceType: new ec2.InstanceType(config.hostInstanceType), // m5.metal
      machineImage: ec2.MachineImage.latestAmazonLinux2023(),
      securityGroup: hostSg,
      role,
      userData,
      requireImdsv2: true,
      blockDevices: [
        {
          deviceName: "/dev/xvda",
          volume: ec2.BlockDeviceVolume.ebs(200, {
            volumeType: ec2.EbsDeviceVolumeType.GP3,
          }),
        },
      ],
    });

The host-agent binary is your code — the one component in this whole series that is not off-the-shelf AWS or Firecracker. We install it here as a release artifact pulled from the same bucket that holds agent images (under a host-agent/current/ prefix), and run it under systemd with Restart=always so a crash never leaves a host silently unable to accept placements. What it actually does on boot is the next section.

The Auto Scaling Group

Now the fleet itself. The ASG spreads hosts across the private subnets from Part 1 (and therefore across availability zones), and its min/desired/max come from config so you can run one host for a demo or scale the ceiling for real load without touching code.

    const fleet = new autoscaling.AutoScalingGroup(this, "HostFleet", {
      vpc,
      vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },
      launchTemplate,
      minCapacity: config.fleetMinHosts,
      desiredCapacity: config.fleetDesiredHosts,
      maxCapacity: config.fleetMaxHosts,
      // Give a bare-metal host time to boot and install Firecracker before
      // the ASG judges its health.
      healthCheck: autoscaling.HealthCheck.ec2({
        grace: cdk.Duration.minutes(5),
      }),
    });

    new cdk.CfnOutput(this, "AsgName", { value: fleet.autoScalingGroupName });
  }
}

Why an ASG rather than a fixed set of instances? Three reasons, all of which pay off in later parts: a host that fails its health check is replaced automatically (Part 7 leans on this for agent rescheduling), the desired count is the single knob demand-based scaling will turn (also Part 7), and rolling a new launch template — a newer Firecracker, a patched AMI — becomes an instance refresh instead of a hand migration.

One bare-metal caveat worth knowing now: *.metal instances are large and comparatively scarce, so scale-out is not instant and can occasionally hit capacity limits in an AZ. For production you would keep a small warm pool so a scale-up event attaches an already-booted host instead of waiting three minutes for one to initialize. We keep it simple here; Part 7 is where scaling gets serious.

Wire the stack into the app, passing it the network outputs from Part 1:

// bin/app.ts
import * as cdk from "aws-cdk-lib";
import { FleetNetworkStack } from "../lib/fleet-network-stack";
import { HostFleetStack } from "../lib/host-fleet-stack";
import { config } from "../lib/config";

const app = new cdk.App();
const env = { region: config.region };

const network = new FleetNetworkStack(app, "EveFleetNetwork", { env });
new HostFleetStack(app, "EveFleetHosts", {
  env,
  vpc: network.vpc,
  hostSg: network.hostSg,
});

And add the host-fleet knobs to the shared config from Part 1:

// lib/config.ts — additions
  /** Bare metal is the only thing that exposes /dev/kvm. */
  hostInstanceType: "m5.metal",
  fleetMinHosts: 1,
  fleetDesiredHosts: 2,
  fleetMaxHosts: 6,
  /** vCPU / memory held back for the host OS + host-agent; the rest is schedulable. */
  hostReservedVcpu: 4,
  hostReservedMemMib: 8192,

The Capacity Model: How Many Agents Fit on a Host

This is the number the whole platform is organised around, so it is worth doing the arithmetic explicitly. An m5.metal is 96 vCPUs and 384 GiB of RAM. We hold some back for the host OS, the CloudWatch agent, and the host-agent itself (hostReservedVcpu / hostReservedMemMib), and everything left over is schedulable — available to hand out to agent microVMs.

Each agent asks for a slice. Using the modest vcpu_count: 2, mem_size_mib: 1024 from the single-VM guide's microVM config, the math is:

schedulable vCPU   = 96 - 4        = 92
schedulable memory = 384 GiB - 8   = 376 GiB

per agent          = 2 vCPU, 1 GiB
vCPU-bound cap     = 92 / 2         = 46 agents
memory-bound cap   = 376 / 1        = 376 agents
→ host capacity    = min(46, 376)   = 46 agents  (CPU is the limit here)

The host-agent tracks this locally: it starts from the schedulable totals, subtracts what each running microVM reserved, and reports the remainder as free capacity in every heartbeat. The control plane (Part 4) never needs to inspect a host directly — it just reads the free-capacity numbers hosts publish and bin-packs new agents onto whichever host has room. That clean split — hosts account for their own capacity, the control plane only places — is what keeps the scheduler simple later.

Which dimension binds (CPU vs. memory) depends entirely on the agent shape. Memory-heavy agents flip the limit; the point is that the host-agent computes it from real reservations rather than guessing, so a mixed fleet of agent sizes still packs correctly.

Registration on Boot

The last thing the host-agent does at startup — once Firecracker is confirmed and it has computed its schedulable capacity — is announce itself. It writes a row into the ${fleetName}-hosts registry table (the table the IAM role grants it access to): its instance ID, availability zone, total schedulable vCPU and memory, and a first heartbeat timestamp. Then it heartbeats every few seconds with current free capacity and a liveness timestamp.

// what the host-agent PUTs on boot, then UPDATEs on each heartbeat
{
  "hostId":        "i-0abc123...",         // partition key
  "az":            "us-east-1a",
  "schedulableVcpu": 92,
  "schedulableMemMib": 385024,
  "freeVcpu":      92,                       // updated every heartbeat
  "freeMemMib":    385024,
  "runningAgents": 0,
  "lastHeartbeat": 1721390400                // control plane treats stale rows as dead
}

Two things make this robust. First, registration is idempotent — a host that reboots and re-registers with the same instance ID overwrites its own row rather than creating a duplicate. Second, the heartbeat timestamp is what the control plane uses to detect a dead host: if lastHeartbeat goes stale, the host's agents are considered lost and get rescheduled, no matter whether the box crashed, was terminated by the ASG, or just lost the network. We are only writing the producer side of this contract in Part 2; the consumer — the scheduler that reads these rows to place and reschedule agents — is Part 4.

Deploy It

The host fleet stack depends on the network stack's outputs, so a single deploy of both (in dependency order) is enough:

cdk deploy EveFleetNetwork EveFleetHosts

Give the bare-metal hosts a few minutes to initialize. Then confirm the fleet is real:

# The ASG reached its desired count.
aws autoscaling describe-auto-scaling-groups \
  --query "AutoScalingGroups[?contains(AutoScalingGroupName,'HostFleet')].Instances[].{Id:InstanceId,State:LifecycleState,AZ:AvailabilityZone}" \
  --output table

# SSM onto one host and check the daemons — no SSH, exactly as designed.
aws ssm start-session --target <instance-id>
firecracker --version
systemctl is-active eve-host-agent

If eve-host-agent is active and your registry table has a row per instance, the fleet is up and announcing itself. There are no agents on it yet — that is the next two parts — but the machines are ready and the control plane will be able to find them the moment it exists.

Tearing Down

Same reverse-order rule as Part 1 — the host fleet comes down before the network it sits in:

cdk destroy EveFleetHosts
# then, if you're done entirely:
cdk destroy EveFleetNetwork

Because bare-metal hosts bill by the second and are the dominant cost in this whole build, do not leave a desiredCapacity of 2 running overnight while you are still just following along. Drop fleetDesiredHosts to 0 (or destroy the stack) between sessions.

What's Next

The fleet exists: bare-metal hosts booting under an Auto Scaling Group, each with Firecracker installed, a host-agent running under systemd, a least-privilege role, and a capacity model that already knows how many agents each box can hold — all announcing themselves into a registry.

In Part 3 we give them something to run. We write the build-rootfs.sh that turns an ordinary eve agent directory into a bootable ext4 rootfs, stand up the versioned S3 artifact bucket the hosts pull from (the one the IAM role above already points at), and look at how Firecracker snapshots turn a cold multi-second boot into a warm sub-second one. After that, Part 4's control plane can finally place an image onto a host and watch an agent come to life.

If you skipped the single-host version, Running an eve agent in a Firecracker microVM on AWS is the foundation this part scales up — it is worth booting one microVM by hand before running a fleet of them.