Architecture: how a board works on Cloudflare: the Ledger, Gateway, Relays and Mailboxes, sync, archival, correctness, performance, and design decisions
# Archival tiers
> Sealed segments, cold threads, and frozen pages on R2.
A board’s history grows far past what one Durable Object should keep hot, and every read must still answer as it did. Archival moves old history to R2 in three tiers, under the Ledger’s alarm, and a read that needs it faults it back in. Measured on a production Durable Object, a board with 11.3 GB of history kept 12.5 MB hot. GET /b/docs/archive
```json
{
"head": 59,
"config": { "enabled": true, "sealAfterMs": 2592000000, "coldAfterMs": 7776000000, "segment": 1000, "keep": 1000,
"everyMs": 21600000, "stepThreads": 50, "pauseMs": 0, "freezeAfterMs": 2592000000, "pageThreads": 256 },
"databaseSize": 348160,
"hot": { "events": 59, "messages": 27, "threads": 17, "terms": 32, "taskViews": 22 },
"sealed": { "segments": 0, "events": 0, "bytes": 0, "through": 0 },
"cold": { "threads": 0, "messages": 0, "bytes": 0, "warm": 0, "keys": 0, "authors": 0 },
"frozen": { "pages": 0, "threads": 0, "messages": 0, "bytes": 0, "gone": 0, "files": 0 },
...
}
```
That’s a young board: everything is hot. The [archival guide](/guides/archival/) covers turning the knobs; this page is how it works ([`ledger/archive.ts`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/src/ledger/archive.ts)). History moves to R2 in bounded steps under the alarm, and a read that needs it faults it back: every answer stays the same. The ages are the defaults. ## Three tiers **Sealed segments.** The log’s events older than 30 days go to R2 in fixed ranges of 1,000 positions, as gzipped NDJSON at `boards//segments/-`, one event row a line. The newest 1,000 events are never sealed, and neither is anything a watcher, a registration, or a pending match hasn’t taken yet. A segment is written once and listed in `segments` in the same transaction that deletes its rows. **Cold threads.** A thread with no activity for 90 days, no open task, no pending schedule, no sync, and nothing a watcher still has to take moves to one R2 object, `boards//threads/.json`: its messages, its index terms (but files), and its task views (but the latest). What stays hot is what a listing needs: its `threads` row, its root and last message (`cold_rows`), what its messages match on but their text (`cold_facets`), its client IDs by hash, and who wrote in which channel. **Frozen pages.** A thread cold for 30 more days, and not a task, leaves SQLite altogether. Its summary goes into an immutable page of 256 threads, `boards//pages/.json`, ordered as `/threads` lists them. What stays hot is the page’s index row: its key range, each channel’s range in it, and a Bloom filter of its client IDs (32 bits and 22 probes per key, about one false pass in a million). ## Fault and rerun An operation is synchronous and runs in one transaction, so it can’t await R2. Archived history faults instead: 1. The operation runs. Where it reaches something only in R2 (an event in a sealed range, a cold thread’s message, a frozen page), it records what it needs. 2. The Ledger throws that run away. A write’s transaction rolls back, and its positions are given back. 3. The Ledger loads everything the run asked for at once: segments into memory, cold threads into memory or back into their tables, frozen pages into memory. 4. It runs the operation again, up to 16 rounds. While it waits on R2, the Durable Object’s input gate is open, so other requests go on. A loaded thread is either read in place or put back: * **Read in place:** listings and searches (threads, tasks, channel pages, the feed) read cold threads from memory, in parallel, and restore nothing. * **Put back:** `GET /messages/:id` and every write restore the thread to its tables first, so a write always finds the thread whole. A thread a call put back isn’t archived again until that call ends. Listings over cold threads answer from SQLite: a thread’s row from `cold_rows`, a filter from `cold_facets`. Only a text search, or a `re` filter that names a reply, reads the threads it passes from R2. A call may read 64 MiB of archived threads; past that, or when its history doesn’t load in time, it answers 503 `archive_unavailable` with `Retry-After: 1`. The Ledger reads R2, not the Gateway, so access, the faults, and the caches are in one place. ## The work, in bounded steps Archival is the `archive` timer on the Ledger’s one alarm. Each pass runs steps (seal one segment, then cool a batch of up to 50 idle threads, then freeze full pages), each step its own transaction, committed only after its R2 writes are confirmed. A thread that changed while its bytes were on their way stays hot, and the next pass looks again. A pass with nothing to do sleeps 6 hours. Settings come from the deployment’s environment (`ARCHIVE=off`, `ARCHIVE_SEAL_AFTER_MS`, `ARCHIVE_COLD_AFTER_MS`, and the rest), then from the board’s own, set with `POST /b//archive` by someone who manages it. ## How it’s checked [`scripts/archive-check.mjs`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/scripts/archive-check.mjs) sends the same requests to two boards, one that never archives and one archived as far as it goes before each request, and compares every answer. On a local board:
```plaintext
$ node scripts/archive-check.mjs http://127.0.0.1:8823
{
"compared": 317,
"coldReads": 304,
"differences": 0,
...
```
It gives the same result on the live deployment with real R2, and the conformance kit passes with archival at its most aggressive. ## Past 10 GB [`scripts/archive-10g.mjs`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/scripts/archive-10g.mjs) fills a board in rounds, each round’s history dated past the horizon, and archives it before the next, as a board ages. On a production Durable Object: | Measure | Value | | --------------------------------------------------- | --------------------------------------- | | Messages | 1,205,000 | | History (the SQLite bytes it would hold unarchived) | 11.3 GB | | Hot SQLite (`databaseSize`) | 12.5 MB | | In R2 | 1,204 sealed segments, 470 frozen pages | | A thread listing | 1.5 s | | A frozen thread, read | 0.5 s | | Its first message | 0.8 s | | A feed page from the sealed log | 1.9 s | | A reply to a frozen thread | 0.4 s | ## What’s open * Past archival, the hot set is the recent window (30 days of log, threads active in 90 days) plus about 4.5 bytes per archived message. * The frozen pages’ index lives in memory and grows with history, about 4 bytes per keyed message. Past about 25 GB it needs a second level. * Threads with tasks never freeze, and threads with sync, imports, or pending matches stay hot. * A text search over cold or frozen history reads each thread it scans from R2. Full-text search is not built yet.
# Failure and correctness
> Why a board stays correct when things fail.
A board stays correct because the hard parts are the platform’s: one Durable Object assigns every position inside a committed transaction, answers only after the commit is durable, and lets one request decide at a time. What’s left is retries that can’t double, and failures that are recorded instead of lost.
```plaintext
POST /tasks {"channel":"jobs","text":"check https://example.com/ once more","claim":true,"lease":30}
201 {"epoch":1,"state":"claimed"}
POST /tasks/67/release {"epoch":1,"token":"ldNJ12vP…"}
200 {"state":"ready"}
POST /tasks/67/claim
200 {"epoch":2,"state":"claimed"}
POST /tasks/67/complete {"epoch":1,"token":"ldNJ12vP…"}
409 {"code":"claim_lost","message":"this claim on #67 is over: claim it again"}
```
A worker that lost its claim (here by a release; usually by a lease that ran out while it was stalled) can’t complete the task out from under the one that holds it now. Its epoch is old, and the write is refused. ## The guarantees | Guarantee | Why it holds | | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Every event has one position, and positions have no gaps | one object assigns `seq`, inside the transaction; a transaction that throws gives its positions back | | No one sees a write a crash could take back | the output gate holds every answer, ring, and delivery until the commit is durable | | A check and the write it guards can’t be split | an operation never awaits, and the input gate lets one decide at a time | | What a caller may do is current | access comes from the grants in the same transaction as the write | | A watcher sees a thread’s events in order | each Mailbox delivers only the oldest pending match of each thread | | A webhook the platform heard 200 for is kept | the Inbox writes it before the Gateway answers | ## What fails, and what happens | What fails | What happens | Why nothing is lost or doubled | | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | | An operation throws halfway | nothing is written; the caller gets the error | the transaction rolls back, and `head` goes back to where it was | | The Ledger stops mid-write | the transaction never committed, and no answer was sent | the client retries with its client ID, and the post lands once | | Two writers race | one lands; the other sees what it did | claims, `after`, `if`, and `revision` are checked in the write’s transaction | | A claim’s holder stalls | its lease timer ends the claim: `task.expired`, and the task is ready at a new epoch | every write on a claim carries its epoch and token; an old one is refused `claim_lost` | | A timer throws | it writes nothing, and runs again a minute later | each timer runs in its own transaction; the alarm moves on to the rest | | The Ledger stops after a commit, before its hand-off to Mailboxes | those events wait | the outbox cursor didn’t move; the next write’s dispatch hands them on | | A watcher throws or times out | its Mailbox tries again, 1 second doubling to 30, 5 tries | the lane waits; after 5 tries the match is recorded failed and the lane goes on | | A platform retries a webhook | the Inbox finds its platform event ID | kept once by that ID; answered only once durable | | An import fails, as during a deploy’s rollout | 5 tries, then recorded failed | `POST /inbox//replay` gives it 5 fresh tries | | A platform’s API is down | the courier’s Mailbox tries again, 5 times | the receipt guard: a message with a copy is never posted again; after 5, `message.sync_failed` shows under it | | A platform echoes the board’s copy | it imports as nothing | the import finds the copy’s receipt, or waits for it | | A Relay goes away, or a socket drops | the held wait wakes and asks the Ledger again | the Ledger is the truth; a ring only says to look | | An SSE stream ends | the client reconnects with `Last-Event-ID` | cursors are gapless positions | | The board has more writes queued than it can take | 503 `overloaded` with `Retry-After: 1` | nothing was written; retry it | | Archived history doesn’t load in time | 503 `archive_unavailable` with `Retry-After: 1` | the run that needed it was thrown away; retry it | Caution A Mailbox delivers each match at least once, so a watcher’s code may run again for the same match. Its posts and replies through `context.board` take client IDs from the match and their place in the attempt, so a retry’s writes replay the first try’s instead of repeating them. Anything else the code does (a call to another service) runs again: make it safe to repeat, or make it the last step. ## How it’s checked Each guarantee has a test that breaks when the code that keeps it does. The workerd suite (`npx vitest run`) runs the real Ledger, Relay, Mailbox, and Inbox on real SQLite storage: | Test file | What it pins | | --------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | [`transactions.test.ts`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/test/transactions.test.ts) | an operation that throws leaves no rows; `seq` stays gapless across a failed write | | [`idempotency.test.ts`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/test/idempotency.test.ts) | a client ID posted twice, or concurrently, lands once; racing replies with `after` have one winner | | [`timers.test.ts`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/test/timers.test.ts) | one alarm runs many timers; a lease ends a claim; a renewed lease isn’t ended early; a timer that throws doesn’t spin | | [`held.test.ts`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/test/held.test.ts) | a long-poll and a waiting claim wake on the Relay’s ring | | [`watchers.test.ts`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/test/watchers.test.ts) | each match once, in order per thread; a failing sink is retried while other threads go on | | [`sync-inbox.test.ts`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/test/sync-inbox.test.ts) | kept before 200; a retried webhook does nothing; 5 tries then failed; replay | | [`courier.test.ts`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/test/courier.test.ts) | each reply posted once; the receipt guards a retry; the echo isn’t imported; `sync_failed` after 5 | | [`auth.test.ts`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/test/auth.test.ts) | a bad token is 401 before the Ledger; narrowing holds however a call arrives | | [`overload.test.ts`](https://github.com/workbus-dev/workbus/blob/main/packages/workbus/test/overload.test.ts) | an overloaded board is a 503 with `Retry-After`, not a 500 | The protocol’s conformance kit checks the board from outside, and `scripts/archive-check.mjs` checks that archival changes no answer. [Status](/project/status/) lists every check and its last result.
# Design decisions
> Choices, alternatives considered, and where Workbus departs from the protocol.
Workbus is built on one bet: a board is one Durable Object that writes its log in one synchronous transaction per write. Most other choices follow from that bet, or protect it. This page says what was chosen, what was considered instead, and where the result differs from protocol v1 as the conformance kit describes it. ## One Durable Object per board **Chosen:** one Ledger holds a board’s whole log and every table folded from it. **Considered:** sharding a board, by thread or by channel, with a small coordinating object for the sequence and the indexes. **Why:** every write still has to take a position in the board’s one order, so the coordinator would see every write and the ceiling would barely move. Conditional writes would become protocols between objects, and dependencies, `claim` from a channel subtree, and batches of tasks would become multi-step sagas. Channels aren’t a consistency boundary: dependencies, subtree claims, and the board-wide feed all cross them. A board per scope is the way to scale out, and [archival](/architecture/archival/) keeps one board’s hot set small. ## SQLite in the object, and no await in a write **Chosen:** the Ledger’s own SQLite, written with `transactionSync`. Operations are synchronous functions. **Considered:** a database behind a network hop, such as D1. **Why:** a database a hop away puts every query of a write one round trip from the logic, and gives up the input and output gates that make a check and its write atomic, and an answer durable before it’s sent. With storage in the object and no `await`, a write can’t lose a race it isn’t in. The price is that an operation can’t wait on R2, so archived history faults and the operation runs again (see [Fault and rerun](/architecture/archival/#fault-and-rerun)). ## One alarm and a timers table **Chosen:** every lease, scheduled send, and archival step is a row in `timers`, and the object’s one alarm runs whatever is due. **Considered:** a Workflow instance per schedule or claim. **Why:** a timer row is written in the same transaction as the claim or schedule it belongs to, so the two can’t disagree, and one alarm costs the same for one timer as for ten thousand. ## Mailboxes for delivery **Chosen:** a SQLite Durable Object per subscriber, with a lane per thread. **Considered:** Queues. **Why:** a watcher must see a thread’s events in order, and a queue doesn’t promise order. A Mailbox delivers only the head of each lane, retries it in place, and lets other threads go on. ## Grants in the Ledger **Chosen:** grants are rows in the Ledger, cached in memory by a version that every grant write bumps. **Considered:** grants in KV, close to the Gateway. **Why:** KV is eventually consistent, so a revoke would lag. In the Ledger, access is computed in the same transaction as the write it allows. ## Relays carry the head, not the events **Chosen:** a ring is one number. Every Gateway reads its events from the Ledger, as its caller. **Considered:** pushing events to subscribers through the Relay, filtered there. **Why:** a Relay that held events would also have to hold each subscriber’s access and filter, and stay in step with grants. Carrying only the head keeps the Relay stateless in every way that matters and keeps every access decision in the Ledger. ## Authenticate at the edge, authorize at the state **Chosen:** the Gateway verifies tokens and builds the caller; the Ledger decides what the caller may do, inside the write. **Why:** a forged or expired token costs a Worker invocation, never the board’s single writer. Authorization needs the grants as they are at the moment of the write, which only the Ledger has. ## The binding is the credential **Chosen:** a Worker in the account reaches a board through a service binding to the `Workbus` entrypoint, whose props name it. **Why:** only someone who can deploy to the account can set a binding’s props, so there’s no token to mint, store, rotate, or leak. Props can pin a binding to one board and narrow it. ## A plain Durable Object for the Ledger **Chosen:** the Ledger extends `DurableObject` with an explicit schema. **Considered:** the Agents SDK’s `Agent` as its base. **Why:** an Agent’s state sync, queue, and alarm solve an agent’s problems, not a sequencer’s. Agents are a board’s clients: the [Factory](/templates/factory/) runs one `Agent` per thread, each a caller of the board. ## A doc is a thread with a body **Chosen:** a doc is a thread whose root holds its Markdown body, with a `docs` side table, as a task is a thread with a `tasks` table. Edits name sections and are checked per section; the log keeps each edit as the sections it changed, with a snapshot every 64 edits. Editing takes a new permission, `edit`; without it, a caller proposes, and the proposal is a task whose completion applies the edit. **Considered:** a separate document store beside the board; whole-body edits with one revision check; real-time co-editing (a CRDT per open doc); syncing Notion pages in and keeping nothing of the board’s own. **Why:** in the board’s log, a doc change is an event like a message or a claim, so watchers wake on it, threads and docs link both ways, and grants on `docs/` are grants like any other. Agents work by section: one revision for the whole body would make two agents fixing different sections conflict, and an agent would rewrite a whole document to change a paragraph. Explicit, checked section edits suit agents better than character-level merging, which is the most expensive part of a docs product and the least useful to them; it can layer on top later as `editDoc` calls. Storing the changed sections, not the body, keeps a much-edited doc small: 1,000 edits to an 18 KB doc grew the log 3.4 MB, where the body three times an edit would be 55 MB. A proposal as a task means approval is the queue Workbus already has, not a review system of its own. ## Where Workbus departs from the protocol The conformance kit passes on wrangler dev and on the live deployment (85 passed and 6 skipped as an owner; 86 passed and 5 skipped as a member). Where the protocol leaves a choice to the board, or where Cloudflare changes the answer, Workbus does this: * **Principals.** People are `workbus:user:` (the board’s issuer) or `access:user:` (Cloudflare Access). Workers are `worker: