Most small businesses connect their shop, their inbox, their accounting and their customer records with scripts or automation tools. That works until one of those tools has a bad day. Then an order goes missing, or arrives twice, and nobody notices until a customer does.

The fifth Entuit demo is about that problem. pipeline.entuit.com is a working integration pipeline for a made-up wholesale business. You send it events, break the tools it delivers to, and watch it cope: retries with backoff, a circuit breaker per destination, duplicates delivered once, and a dead-letter list for anything that never got through. Every event comes with its trace, so you can see where the time went.

The four other demos show what a model can do. This one shows the engineering around it. This post covers the stack, what each piece does and why we chose it, and how the pipeline is designed and built.

What the Demo Does

The business is Northbound Supply. Three made-up tools send it events, and three must receive them:

Sources (send events)	Destinations (receive them)
The web shop: a new order	Ledgerly, accounting: orders and invoices
The support inbox: a customer email	Fieldbook, a CRM: customers and their history
A supplier: an invoice	Team chat: anything a person should see today

Each event is validated, classified by a small AI model (is it urgent, and what is it about), routed, and delivered to each destination that needs it. An angry email about a damaged pallet goes to the CRM and to team chat. A routine order goes to accounting and the CRM.

Each destination has a control: healthy, slow (2 to 8 seconds), flaky (fails 40% of the time) or down. Set accounting to down, send an order, and the timeline shows the attempts, the breaker opening, the event going back on the queue, and, once you set accounting back to healthy, a delivery to accounting only, since the CRM already has it.

Each visitor also gets a webhook URL, so you can send your own event from a terminal:

curl -X POST https://pipeline.entuit.com/hooks/<session>/shop \
  -H "content-type: application/json" \
  -d '{"orderId":"A-1050","customer":"Your Name","total":99.50}'

A body that doesn't match the schema gets a 400 with the reason, before anything is queued.

The Stack

Technology	What it does	Why we used it here
Effect	A TypeScript library for programs that can fail: typed errors, retries, timeouts, concurrency, dependency injection and tracing	The pipeline is almost entirely failure handling. Effect makes each failure part of a function's type and makes retry and timeout policies one line each
Cloudflare Workers	Runs code at Cloudflare's edge, close to each visitor	No servers to run, fast cold starts, and every other piece below is a binding away
Cloudflare Queues	A durable queue between Workers, with retries, delays and a dead-letter queue	The hand-off from the webhook to the pipeline. A Worker that dies mid-event loses nothing
Durable Objects	A single-threaded object with its own storage, addressed by name	One per visitor holds the event log, the destinations' settings and the circuit breakers, so every queue consumer sees the same state. It also holds the page's WebSockets
Workers AI and AI Gateway	Open models on Cloudflare's GPUs, called through a gateway with analytics and a rate limit	One small classification per event, a few Neurons each. The gateway adds a daily cap that doesn't depend on our code
Alchemy	Infrastructure as code, written as an Effect program	The Workers, queues, objects and Turnstile widget are declared in the same TypeScript as the code that uses them, with no Wrangler config
Foldkit	A front-end framework on Effect, in the Elm style: one model, an update function, commands and subscriptions	The page is Effect too, sharing the Worker's schemas. A live dashboard is a state machine, which is what this style is for
Turnstile	Cloudflare's bot check, usually invisible	Once per session, before the first event
Bun	Package manager, script runner and test runner	Fast, and the same tool as our other repos. In production the code runs on Cloudflare's runtime, not Bun

The Architecture

Browser (Foldkit page) ── WebSocket, POST ──┐
curl ────────────────────── webhook ────────┤
                                            ▼
                                       Page Worker
                                            │ service binding
                                            ▼
                                       Api Worker
                                 validate, check, count
                                            │
                                            ▼
                                      Queue: events
                                            │
                                            ▼
                               Queue consumer: the pipeline
                               classify ─▶ Workers AI
                                           (via AI Gateway)
                               deliver  ─▶ simulated tools
                                            │
                         retries used up    │ progress, spans
                                ▼           ▼
                 Queue: dead letters ─▶ Session Durable Object
                                        (one per visitor)
                                            │
                                            ▼ live updates
                                         Browser

There are two Workers. The page Worker serves the Foldkit build and forwards /hooks/* and /session/*, WebSocket upgrades included, to the Api Worker over a service binding. So the page and the API share an origin, with no CORS, and the Api Worker has no public address of its own. The Api Worker validates requests, hosts the Session and Ledger Durable Objects, and runs both queue consumers.

There is no database. Everything a visitor creates lives in their Session object, which deletes itself by alarm 24 hours after their last activity. A second object, the Ledger, holds the daily budgets for the whole demo and the saved classifications.

The destinations are simulated inside each visitor's Session: a function that reads the destination's health setting and then waits, fails or succeeds. That keeps visitors isolated, makes every failure reproducible and costs nothing. The pipeline calls them through an Effect service called Destination. In a real project, the same pipeline code would get a Destination that calls the real API.

The Pipeline, in Effect

The pipeline lives in its own package with no Cloudflare imports. It depends only on Effect services (Destination, Classifier, Breakers, Deliveries, SavedClassifications), which the Worker provides at runtime and the tests provide in memory. Here is the whole of it for one event:

export const processEvent = (event: Event) =>
  Effect.gen(function* () {
    const key = yield* idempotencyKey(event);
    const classification = yield* classify(event);
    const deliveries = yield* Effect.forEach(
      route(event, classification),
      (to) => deliver(event, key, to),
      { concurrency: 3 },
    );
    return { key, classification, deliveries };
  }).pipe(Effect.withSpan("event", { attributes: { kind: event.kind } }));

Key the event, classify it, work out where it goes, and deliver it to each destination in parallel, at most three at a time. Each part is worth a closer look.

Schemas Shared by the Worker and the Browser

Every event is an Effect Schema. The webhook route, the queue consumer and the browser all decode with the same schemas, and so are the messages on the WebSocket. If one side changes a message in a way that breaks the other, the type check fails before anything ships. The schema also does the edge validation: an order ID must look like A-1042, a total must be positive, and text fields have length limits.

Typed Errors

Each way a delivery can fail is a tagged error:

/** The destination took longer than the pipeline waits. Retried. */
export class Timeout extends Schema.TaggedError<Timeout>()("Timeout", { after: Schema.String }) {}

/** The destination answered with an error status. 5xx is retried, 4xx is not. */
export class Rejected extends Schema.TaggedError<Rejected>()("Rejected", {
  status: Schema.Number,
  reason: Schema.String,
}) {}

/** The destination's breaker is open; nothing was sent. */
export class CircuitOpen extends Schema.TaggedError<CircuitOpen>()("CircuitOpen", { until: Schema.Number }) {}

export const isRetryable = (error: DeliveryError): boolean =>
  error._tag === "Timeout" || (error._tag === "Rejected" && error.status >= 500);

These errors are in the type of every function that can produce them, so the compiler makes each caller handle or pass on each one. Nothing is lost in a catch (e: unknown).

Delivery: Time Limit, Retries, Breaker

One delivery to one destination:

export const TIME_LIMIT = "3 seconds";
export const RETRIES = 4;
/** 200 ms, 400 ms, 800 ms, 1.6 s, each scaled by a random 0.8 to 1.2. */
export const backoff = Schedule.exponential("200 millis").pipe(Schedule.jittered);

const attempt = Effect.gen(function* () {
  yield* guard(to); // ask the breaker; fails with CircuitOpen if it is open
  return yield* destination.call(to, event, key).pipe(
    Effect.timeoutOrElse({
      duration: TIME_LIMIT,
      orElse: () => Effect.fail(new Timeout({ after: TIME_LIMIT })),
    }),
    Effect.tapError(() => report(to, false)),
    Effect.tap(() => report(to, true)),
  );
}).pipe(Effect.withSpan("attempt"));

const outcome = yield* attempt.pipe(
  Effect.retry({ schedule: backoff, times: RETRIES, while: isRetryable }),
  Effect.result,
);

The time limit, the retry policy and the breaker are each one line, and they compose: a slow destination times out, the timeout is retryable, the retries back off with jitter so a burst of events doesn't hammer a struggling tool in step, and each failure is reported to the breaker. A 503 is retried. A 4xx would not be: retrying a request the destination has refused only repeats the refusal.

Before calling, deliver checks whether this destination already has this event, and skips it if so. After a success it records the receipt.

The Circuit Breaker

The breaker is a set of pure functions over a small state type:

export type BreakerState =
  | { readonly _tag: "Closed"; readonly failures: ReadonlyArray<number> }
  | { readonly _tag: "Open"; readonly until: number }
  | { readonly _tag: "HalfOpen"; readonly probeAt: number };

While it is closed, calls go through and failures are counted. Five failures within 30 seconds open it, and calls then fail at once with CircuitOpen, without touching the destination. After 30 seconds, one call is let through (half-open): success closes the breaker, failure opens it again. If that probe never reports back, because its Worker died, another call is let through after 10 seconds.

Being pure, the functions are tested directly, with no clocks or mocks. The state for each destination lives in the visitor's Session object, which runs admit and record itself, so each change is one atomic step and every queue consumer sees the same breaker.

Duplicates Delivered Once

Each event gets an idempotency key: the source's own ID where it has one (an order number, a supplier and invoice number), otherwise a SHA-256 of the body. The Session records which destinations have each key. A retried or repeated message is skipped by the destinations that already have it, and the timeline shows "duplicate". Send the same order twice and it reaches accounting once.

Classification With a Time Limit and Two Fallbacks

The classification decides where an event goes and adds a one-line summary:

const chain = (event: Event) =>
  ask(event, PRIMARY, "primary").pipe(
    Effect.catch(() => ask(event, FALLBACK, "fallback")),
    Effect.catch(() => Effect.succeed(byRules(event))),
  );

PRIMARY is Llama 3.2 3B and FALLBACK is Llama 3.2 1B, both on Workers AI and each with a 2-second limit. If both fail, keyword rules still route the event, and the timeline says the rules answered. The model gets a system prompt that treats the event's text as data, a JSON schema for its answer, a cap of 80 tokens and a temperature of 0, so it answers with a priority and a short summary rather than a paragraph.

Classifications are saved in the Ledger by a hash of the text. Every unedited sample has been seen before, so most events cost no model call at all. An edited one costs a few Neurons.

The Queue: Retries That Don't Hold a Worker Open

The retries above happen within one run of the queue consumer, and they give up within seconds. If a destination is still failing after that, the consumer doesn't wait. It puts the message back on the queue with a delay: a few seconds for a timeout or a 503, or until the breaker's wait ends if the breaker is open. On the next run, the destinations that already have the event skip it, so only the failed one is tried again.

After six runs, the queue hands the message to the dead-letter queue, whose own consumer marks the event as dead-lettered in the visitor's log, with the reason. From there, Replay sends it through again once the destination is back.

Tracing

Every step is wrapped in Effect.withSpan. A small custom Tracer records each finished span (name, start, end, attributes, outcome) and the consumer streams them to the visitor's Session a few times a second while the event is running. The page draws them as a waterfall: the event, the classification and each model call, each delivery and each attempt, with the failures marked and named. Cloudflare's own Workers tracing stays on as well.

The Infrastructure, in Alchemy

Alchemy describes the resources and the code that uses them in one Effect program. The stack file is short:

export default Alchemy.Stack(
  "EntuitPipeline",
  { providers: Cloudflare.providers(), state: Cloudflare.state() },
  Effect.gen(function* () {
    const stage = yield* Alchemy.Stage;
    const prod = stage === "prod";
    const api = yield* Api;
    if (prod) {
      const widget = yield* Cloudflare.Turnstile.Widget("Turnstile", {
        domains: ["pipeline.entuit.com"],
        mode: "managed",
      });
      yield* api.bind("Turnstile", {
        env: {
          TURNSTILE_SITEKEY: widget.sitekey,
          TURNSTILE_SECRET: widget.secret,
          TURNSTILE_HOSTNAME: "pipeline.entuit.com",
        },
      });
    }
    const site = yield* Cloudflare.Website.Foldkit("Pipeline", {
      rootDir: "apps/web",
      main: "src/worker.ts",
      assets: { runWorkerFirst: ["/hooks/*", "/session/*"] },
      env: { API: api },
      domain: prod ? "pipeline.entuit.com" : undefined,
    });
    return { url: site.url };
  }),
);

The Turnstile widget is a resource, and its secret goes straight to the Api Worker as an encrypted binding. Nobody copies it by hand. The page Worker gets the Api Worker as a service binding by being handed the resource.

The Api Worker itself is a class whose body is an Effect program. Its bindings are declared by using them:

const sessions = yield* Session;
const events = yield* Events;
const queue = yield* Cloudflare.Queues.WriteQueue(events);
const perAddress = yield* Cloudflare.Workers.RateLimit("ADDRESS_LIMIT", {
  namespaceId: 4105,
  simple: { limit: 20, period: 60 },
});

yield* Cloudflare.Queues.consumeQueueMessages(
  events,
  { batchSize: 10, maxRetries: 5, retryDelay: "5 seconds", deadLetterQueue },
  (stream) => Stream.runDrain(Stream.mapEffect(stream, handle, { concurrency: 10 })),
);

Yielding the Session class binds the Durable Object namespace. Yielding the queue and wrapping it in WriteQueue binds it for sending. consumeQueueMessages registers this Worker as the queue's consumer, with its batch size, retries and dead-letter queue. At deploy time Alchemy reads these and configures Cloudflare; at runtime the same lines return the clients. There is no separate config file to drift out of step with the code.

Stages come for free. alchemy dev runs the whole stack on one machine, with Workers, Durable Objects and Queues emulated and no Cloudflare account involved. Preview stages get their own copies of everything, and prod is the only stage with the custom domain and a real Turnstile widget. Every other stage uses Cloudflare's test keys. Alchemy keeps its record of what it deployed in a small state store on the account, so deploys work from any machine.

The Page, in Foldkit

The page is a Foldkit app: one Model, an update function that turns each message into a new model and any commands to run, and subscriptions for things that happen over time. The visitor's WebSocket is a managed resource, open while the model wants it. Everything the session pushes (a snapshot, an event's progress, a breaker opening) arrives as a message and goes through update, so the page never guesses at server state.

Sending an event, changing a destination's health and replaying a dead letter are commands that POST to the Api Worker. Their results come back over the WebSocket like everything else. The bot check is a subscription too. While the session hasn't passed, it renders an interaction-only Turnstile widget outside the app's root and turns the token into a message.

Because the page is written in Effect with the Worker's schemas, the same types describe an event in the browser, on the wire and in the pipeline.

Guards and Cost

A public demo that calls a model needs limits. These are the guards:

Turnstile, once per session. Every POST except the check itself gets a 403 until the session has passed. The Api Worker redeems each token at Cloudflare's siteverify and checks the action and the hostname.
A per-address rate limit on every POST, using a Workers rate-limit binding, checked before any Durable Object is called. It is approximate by design, counted per Cloudflare location, so it slows a script down rather than capping it exactly. A zone rule on entuit.com (100 requests in 10 seconds per address) is the hard per-address limit.
Per-session and daily caps: 60 events per session, 3,000 events a day across the demo, 200 model calls and 1,500 Neurons a day, all counted in the Ledger. Past the model cap the keyword rules answer, so the demo keeps working.
A pause switch: a variable on the Api Worker that can be changed in the Cloudflare dashboard without a deploy. While it is on, new events are refused and waiting ones are set aside for replay without any work being done.

At every daily cap, every day for a month, the whole demo would cost about $0.60 USD. A normal month costs cents.

Testing

The pipeline and schema packages have 30 unit tests. The pipeline's run on Effect's test clock, so a 30-second breaker wait or a chain of backoff delays takes no real time, and the whole set runs in under a fifth of a second.

The stack tests deploy the whole thing, both Workers, both Durable Objects and both queues, under Alchemy's local emulation, and drive it like a visitor: connect the WebSocket, pass the bot check, send an order, take a destination down and bring it back, replay an event, hit the session cap and the rate limit. They run in about 50 seconds with no Cloudflare account.

Using This for Real

Everything here except the simulated destinations is what a real integration needs. To connect a real shop to real accounting software, the pipeline package stays as it is. It gets a Destination service that calls the accounting API, and a webhook route that checks the shop's signature. The retries, the breaker, the duplicate handling, the dead letters and the traces come with it.

If your business moves data between tools by hand, or with a script that breaks quietly, book a call and we can talk about what that would look like for you.