> Blog Post

Running a Fleet of Firecracker microVMs for eve.dev Agents, Part 4: The Control Plane

awsaws-cdkfirecrackereveagentstypescriptlambdadynamodbai-infrastructure

Three parts in, we have every piece except the one that makes it a platform. Part 2 gave us hosts that announce their free capacity into a registry. Part 3 gave us bootable agent images sitting in S3. What is missing is the thing in the middle: something that takes "deploy my-agent version a1b2c3d," decides which host has room for it, records that decision, and tells that host to boot the microVM. That is the control plane, and it is what turns a pile of machines and images into a system you can deploy to.

By the end of Part 4, a single HTTP call — POST /agents — will place an agent onto a host and bring it up, and a GET will tell you where it landed and whether it is running. This is the part where deploy my-agent finally produces a running agent.

What the Control Plane Owns

Keep the responsibilities narrow and the design stays simple. The control plane owns exactly four things:

  • The API — a small HTTP surface: deploy an agent, destroy one, list them, get one's status.
  • The registry — three DynamoDB tables that are the system's source of truth: which hosts exist and how much they have free, which agents are supposed to be running, and where each one is actually placed.
  • Placement — the algorithm that, given an agent's resource request, picks a host with room and reserves that capacity without racing other placements.
  • Commanding the host — telling the chosen host's host-agent (on the port-7000 management path from Part 1) to boot the microVM from the right image.

Critically, the control plane does not run agents, touch Firecracker, or move network traffic. It decides and records; the host-agents execute; the routing layer (Part 5) carries traffic. That separation is what lets the whole thing be serverless — there is no long-running scheduler process to babysit, just Lambdas that fire when there is a decision to make.

The Registry: Three Tables

Everything hangs off the data model, so start there. Three tables, each with one job.

// lib/control-plane-stack.ts (registry)
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as dynamodb from "aws-cdk-lib/aws-dynamodb";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as apigw from "aws-cdk-lib/aws-apigatewayv2";
import * as integrations from "aws-cdk-lib/aws-apigatewayv2-integrations";
import * as events from "aws-cdk-lib/aws-events";
import * as targets from "aws-cdk-lib/aws-events-targets";
import { config } from "./config";

export interface ControlPlaneStackProps extends cdk.StackProps {
  vpc: ec2.IVpc;
  controlPlaneSg: ec2.ISecurityGroup;
}

export class ControlPlaneStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: ControlPlaneStackProps) {
    super(scope, id, props);
    const { vpc, controlPlaneSg } = props;

    const ddbProps = {
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      removalPolicy: cdk.RemovalPolicy.DESTROY, // tutorial; RETAIN in prod
    };

    // Hosts — the capacity registry the host-agents write to (Part 2).
    const hosts = new dynamodb.Table(this, "Hosts", {
      tableName: `${config.fleetName}-hosts`,
      partitionKey: { name: "hostId", type: dynamodb.AttributeType.STRING },
      ...ddbProps,
    });

    // Agents — desired state: what should be running, at what size and version.
    const agents = new dynamodb.Table(this, "Agents", {
      tableName: `${config.fleetName}-agents`,
      partitionKey: { name: "agentId", type: dynamodb.AttributeType.STRING },
      ...ddbProps,
    });

    // Placements — the binding: which agent is on which host, right now.
    const placements = new dynamodb.Table(this, "Placements", {
      tableName: `${config.fleetName}-placements`,
      partitionKey: { name: "agentId", type: dynamodb.AttributeType.STRING },
      ...ddbProps,
    });
    // Find every agent on a given host — needed to reschedule when a host dies (Part 7).
    placements.addGlobalSecondaryIndex({
      indexName: "byHost",
      partitionKey: { name: "hostId", type: dynamodb.AttributeType.STRING },
    });

    // ...event bus, lambdas, and API below
  }
}

The hosts table is the one Part 2's host-agents already write to — the control plane is its reader. A host row carries its schedulable and free vcpu/mem, its lastHeartbeat, and its private IP (the address the scheduler will call on port 7000). The agents table is desired state — "this agent, this version, this size, should be running." The placements table is actual state — the binding of agent to host. Keeping desired and actual in separate tables is what makes reconciliation in Part 7 tractable: the health loop's whole job is making actual match desired.

EventBridge: Decouple the API From the Work

The naive design has the API Lambda do everything synchronously — receive the request, scan hosts, place, command the host, then respond. That makes the caller wait on a microVM boot and gives you nowhere to retry if a host command fails. Instead, we split it with an event bus. The API does the fast part (validate, record desired state, emit an event) and returns immediately; a scheduler Lambda does the slow part (place and command) off the event.

    const bus = new events.EventBus(this, "FleetBus", {
      eventBusName: `${config.fleetName}-events`,
    });

This buys three things. The API stays fast — POST /agents returns 202 Accepted in milliseconds. Failures get automatic retries — a host command that fails is a Lambda invocation EventBridge will retry, not a dropped request. And the same events (agent.deploy.requested, agent.placed, agent.running, agent.unplaceable) become the audit trail and the hooks Part 7's health loop subscribes to. The bus is the spine the whole lifecycle hangs off.

The Two Lambdas

There are two functions. The API handler is a plain HTTP Lambda — no VPC, it only touches DynamoDB and the bus. The scheduler must live inside the VPC, in the controlPlaneSg from Part 1, because it makes the one call that reaches into private space: the HTTP command to a host-agent on port 7000.

    const env = {
      HOSTS_TABLE: hosts.tableName,
      AGENTS_TABLE: agents.tableName,
      PLACEMENTS_TABLE: placements.tableName,
      BUS_NAME: bus.eventBusName,
    };

    // API handler — fast path. Records desired state, emits an event, returns.
    const api = new lambda.Function(this, "ApiFn", {
      runtime: lambda.Runtime.NODEJS_20_X,
      handler: "api.handler",
      code: lambda.Code.fromAsset("lambda/api"),
      environment: env,
    });
    agents.grantReadWriteData(api);
    placements.grantReadData(api);
    bus.grantPutEventsTo(api);

    // Scheduler — slow path. In the VPC so it can reach host-agents on :7000.
    const scheduler = new lambda.Function(this, "SchedulerFn", {
      runtime: lambda.Runtime.NODEJS_20_X,
      handler: "scheduler.handler",
      code: lambda.Code.fromAsset("lambda/scheduler"),
      environment: env,
      timeout: cdk.Duration.seconds(30),
      vpc,
      vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },
      securityGroups: [controlPlaneSg],
    });
    hosts.grantReadWriteData(scheduler);
    agents.grantReadWriteData(scheduler);
    placements.grantReadWriteData(scheduler);
    bus.grantPutEventsTo(scheduler);

    // Deploy/destroy events drive the scheduler.
    new events.Rule(this, "LifecycleRule", {
      eventBus: bus,
      eventPattern: {
        source: ["eve.fleet"],
        detailType: ["agent.deploy.requested", "agent.destroy.requested"],
      },
      targets: [new targets.LambdaFunction(scheduler)],
    });

    // HTTP API in front of the API handler.
    const httpApi = new apigw.HttpApi(this, "ControlApi", {
      apiName: `${config.fleetName}-control`,
    });
    const integ = new integrations.HttpLambdaIntegration("ApiInteg", api);
    httpApi.addRoutes({
      path: "/agents",
      methods: [apigw.HttpMethod.POST, apigw.HttpMethod.GET],
      integration: integ,
    });
    httpApi.addRoutes({
      path: "/agents/{id}",
      methods: [apigw.HttpMethod.GET, apigw.HttpMethod.DELETE],
      integration: integ,
    });

    new cdk.CfnOutput(this, "ControlApiUrl", { value: httpApi.apiEndpoint });

Wire it into the app alongside the earlier stacks, passing the network handles from Part 1:

// bin/app.ts — additions
const network = new FleetNetworkStack(app, "EveFleetNetwork", { env });
new HostFleetStack(app, "EveFleetHosts", { env, vpc: network.vpc, hostSg: network.hostSg });
new ArtifactStack(app, "EveFleetArtifacts", { env });
new ControlPlaneStack(app, "EveFleetControlPlane", {
  env,
  vpc: network.vpc,
  controlPlaneSg: network.controlPlaneSg,
});

The API Handler

The API handler is deliberately dumb. It never places anything — it records what should be true and lets the scheduler make it so.

// lambda/api/api.ts
export async function handler(event) {
  const method = event.requestContext.http.method;
  const id = event.pathParameters?.id;

  if (method === "POST") {
    // Deploy: record desired state, fire an event, return immediately.
    const { name, version, vcpu = 2, mem = 1024 } = JSON.parse(event.body);
    const agentId = `${name}-${version}`;
    await putAgent({ agentId, name, imageVersion: version, vcpu, mem, desired: "running" });
    await emit("agent.deploy.requested", { agentId });
    return json(202, { agentId, status: "scheduling" });
  }

  if (method === "DELETE" && id) {
    // Destroy: flip desired state, fire an event; the scheduler tears it down.
    await updateAgentDesired(id, "stopped");
    await emit("agent.destroy.requested", { agentId: id });
    return json(202, { agentId: id, status: "stopping" });
  }

  if (method === "GET" && id) return json(200, await getStatus(id)); // joins agent + placement
  if (method === "GET") return json(200, await listAgents());
  return json(405, { error: "method not allowed" });
}

POST returns 202 with an agentId the moment desired state is written — the actual boot happens asynchronously. GET /agents/{id} joins the agent's desired state with its placement row so the caller sees both "supposed to be running" and "running on host i-0abc… since 12:04."

Placement: Bin-Packing, Race-Safe

This is the heart of it. The scheduler receives the agent.deploy.requested event, looks up the agent's size, and finds a host. Two design choices matter. First, bin-packing: among hosts with enough room, prefer the tightest fit — the one that will have the least left over — so agents pack densely onto fewer hosts and the expensive bare-metal boxes stay busy instead of half-empty. Second, and non-negotiable, race-safety: two placements happening at once must never both claim the last slice of a host's capacity.

// lambda/scheduler/scheduler.ts
export async function handler(event) {
  const { agentId } = event.detail;
  if (event["detail-type"] === "agent.destroy.requested") return teardown(agentId);

  const agent = await getAgent(agentId); // { name, imageVersion, vcpu, mem }

  // Candidate hosts with room, tightest-fit first (bin-pack for density).
  const candidates = (await scanHosts())
    .filter((h) => h.freeVcpu >= agent.vcpu && h.freeMemMib >= agent.mem)
    .sort((a, b) => (a.freeVcpu - agent.vcpu) - (b.freeVcpu - agent.vcpu));

  for (const host of candidates) {
    try {
      await reserveCapacity(host.hostId, agent.vcpu, agent.mem); // atomic; throws if raced
      await putPlacement({ agentId, hostId: host.hostId, version: agent.imageVersion, status: "booting" });
      await commandHost(host, agent);
      await emit("agent.placed", { agentId, hostId: host.hostId });
      return;
    } catch (e) {
      if (e.name === "ConditionalCheckFailedException") continue; // lost the race — next host
      throw e;
    }
  }

  // Nobody had room. Part 7 subscribes to this to trigger a scale-out.
  await emit("agent.unplaceable", { agentId });
}

// The capacity reservation IS the lock. The condition makes oversubscription impossible.
async function reserveCapacity(hostId, vcpu, mem) {
  await ddb.update({
    TableName: process.env.HOSTS_TABLE,
    Key: { hostId },
    UpdateExpression:
      "SET freeVcpu = freeVcpu - :v, freeMemMib = freeMemMib - :m, runningAgents = runningAgents + :one",
    ConditionExpression: "freeVcpu >= :v AND freeMemMib >= :m",
    ExpressionAttributeValues: { ":v": vcpu, ":m": mem, ":one": 1 },
  });
}

That ConditionExpression is the whole game. The reservation and the capacity check are a single atomic DynamoDB operation: if another placement got there first and the host no longer has room, the condition fails, the update is rejected, and we simply try the next host. No locks, no scheduler singleton, no oversubscribed hosts — the database enforces correctness under any amount of concurrency. This is what lets placement be a stateless Lambda that can run many copies at once.

Commanding the Host

With capacity reserved and a placement row written, the scheduler tells the host to boot. This is the one call that crosses into private space, which is exactly why the scheduler runs in the VPC under controlPlaneSg — Part 1's security group lets that group reach hosts on port 7000, and nothing else can.

async function commandHost(host, agent) {
  const res = await fetch(`http://${host.privateIp}:7000/microvms`, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({
      agentId: agent.agentId,
      image: `agents/${agent.name}/${agent.imageVersion}/rootfs.ext4`, // S3 key from Part 3
      vcpu: agent.vcpu,
      memMib: agent.mem,
    }),
  });
  if (!res.ok) throw new Error(`host ${host.hostId} rejected boot: ${res.status}`);
}

The host-agent takes it from here exactly as Parts 2 and 3 described: pull that rootfs key from the artifact bucket (cached locally), write the agent's secrets into the microVM's MMDS, and boot Firecracker. When the microVM is up, the host-agent flips the placement row to running and emits agent.running. If the boot fails, it frees the capacity it reserved and emits agent.boot.failed — and because everything went through EventBridge, that failure is a retriable event, not a silent dead agent.

Deploy It and Watch an Agent Land

cdk deploy EveFleetControlPlane

Grab the ControlApiUrl output, then deploy the sample agent you built in Part 3:

curl -sX POST "$CONTROL_API/agents" \
  -d '{"name":"my-agent","version":"a1b2c3d","vcpu":2,"mem":1024}'
# → 202 {"agentId":"my-agent-a1b2c3d","status":"scheduling"}

# A moment later, ask where it landed:
curl -s "$CONTROL_API/agents/my-agent-a1b2c3d"
# → {"agentId":"my-agent-a1b2c3d","desired":"running","status":"running",
#    "hostId":"i-0abc123","version":"a1b2c3d"}

Behind those two calls: the API wrote desired state and emitted an event; the scheduler bin-packed the agent onto a host and atomically reserved its capacity; the host-agent pulled the image, injected secrets, and booted the microVM; and the placement row flipped to running. You now have an agent alive on the fleet, chosen and placed by the platform rather than by you. What you cannot do yet is send it a request — it has no address the outside world can reach. That is Part 5.

Tearing Down

The control plane comes down before the hosts and network it coordinates:

cdk destroy EveFleetControlPlane

One caveat: destroying the control plane removes the tables, but any microVMs already running on hosts keep running — the host-agents do not stop them just because the scheduler went away. Drain agents first (DELETE /agents/{id} for each, or drop the host fleet's desired count to zero from Part 2) before tearing the control plane down, or you will leave orphaned microVMs on hosts with no registry that remembers them.

What's Next

The platform can now make decisions: an HTTP call records intent, EventBridge carries it, a race-safe bin-packing scheduler picks a host and reserves its capacity, and the host-agent boots the microVM — all serverless, all reconcilable. An agent is running on the fleet.

It just is not reachable. In Part 5 we build the routing layer that fixes that: per-microVM tap devices and NAT on each host so a guest has a real address, an ALB out front, and a per-agent subdomain scheme so a request to my-agent.fleet.example.com finds the exact microVM the control plane placed — wherever it landed. That is the step that turns "an agent is running somewhere" into "you can talk to it."

If you want the ground-level mechanics behind the boot the host-agent performs here, Running an eve agent in a Firecracker microVM on AWS walks the single-microVM version by hand.