Scaling out

A single sithbitd process on SQLite is the zero-config default and the right shape for one operator on one host (see Running a mail server for getting that far). To run multiple instances of the mail services (SMTP/IMAP/POP listeners, account-api, spooler workers) behind a load balancer or an orchestrator, every box below must be ticked — each one is an invariant the single-process default provides for free. The instances need not be identical, either: the same section toggles split a fleet by role — see Role-split topologies below.

The checklist

1. A shared store. [store] kind = "postgres", "aws", or "azure" (postgres is also the Google Cloud shape, against Cloud SQL). SQLite is one process per store, full stop: its write pool is a single connection over a local file, and its blob/watch defaults are process-local. kind = "turso" follows the same one-process rule: it is a local libSQL file, optionally an embedded replica that syncs to a remote Turso/libSQL primary. The replica keeps a synced-to-cloud copy for durability and local-speed reads, but reads still hit the local file (which lags the primary by up to sync_interval_secs), so it does not give the cross-instance lease/queue consistency postgres/aws/azure do — treat turso like SQLite for scaling, not as a shared store.

   kind = "cloudflare" (D1 + Queues + Workers KV + R2) is a true shared store: Cloudflare Queues give cross-instance queue coordination, keyed leases are strict single-statement SQL on D1’s leases table, and IMAP-uid allocation (uidnext) is a server-side atomic UPDATE … RETURNING — D1 executes each statement atomically on its per-database SQLite writer, so N daemons delivering into the same mailbox over one D1 database allocate distinct uids. The historical one-writer delivery caveat (uid allocation was once serialized only by an in-process mutex) was removed 2026-07-19; D1’s lack of a multi-statement transaction no longer constrains scaling, because nothing counter-critical spans statements anymore.

2. A shared blob store. [store.blobs] must point at S3 (which covers GCS via its S3-interop endpoint) or Azure Blob. The local-directory backend only works multi-instance on a shared volume, which is discouraged.

3. The same key files everywhere. credential.key (the mail-password seal), the account API’s JWT key, and any DKIM signing key must be identical on every replica — distribute them as secrets, and back them up: losing credential.key orphans every stored password.

4. IDLE polling on IMAP instances. [imap] watch_poll_seconds = N (e.g. 2–5). Delivery push is in-process; an instance that didn’t do the delivering only learns of new mail by polling. Instances that combine the spooler and IMAP still push their own deliveries instantly. An instance running IMAP with both SMTP roles off adopts 5 s automatically when the setting is left at its 0 default — see Role-split topologies.

5. PROXY protocol or source-IP preservation at the balancer. DNSBL checks and per-client connection limits key on the peer address. Behind an L4 balancer that rewrites sources, either preserve client IPs (e.g. Kubernetes externalTrafficPolicy: Local) or enable proxy_protocol = true in each listener’s server section and the balancer. Never enable it on a listener that clients can reach directly — the preamble is trivially spoofable.

6. Replica-aware limits. max_connections/max_per_peer are enforced per process; the fleet-wide effective cap is the per-instance value times the replica count.

What already just works

• Job queues (chain/relay/DSN/delete) claim atomically under concurrent workers on all backends; jobs are at-least-once and every handler tolerates duplicates.
• Per-recipient SendMail ordering is serialized across instances by a store lease (send/{wallet}), so concurrent spoolers cannot double-send or double-spend stamps on one mailbox.
• POP maildrop exclusivity is a store lease with self-expiry — a crashed session on one instance cannot wedge the maildrop for the rest.
• The reconciler may run on every instance; duplicate re-enqueues are absorbed by the chain worker’s state guards.
• account-api replicas share nonces and credentials through the store; any replica can answer any request.

Known seams

• \Recent (IMAP) is best-effort across instances: two concurrent SELECTs of one mailbox on different instances may both see a message as recent.
• IDLE latency on a poll-fed instance is bounded by watch_poll_seconds, not instant.
• An IMAP session’s selected-mailbox snapshot is taken at SELECT: an idler woken by a cross-process delivery gets its untagged EXISTS, but FETCHing the new message takes a re-SELECT first (NOOP-driven refresh is deferred work).
• Cloudflare leases are strict (since 2026-07-19). Lease acquisition (send/{wallet} SendMail serialization, pop/{wallet} maildrop locks) is the same single-statement compare-and-swap upsert the SQLite/Turso stores run, executed on D1’s leases table — atomic server-side, no TTL floor, expiry a plain integer comparison. The original Workers KV lease (read-then-write, no CAS, ~60 s minimum TTL, eventually-consistent expiry) is retired; every backend’s leases are now strictly atomic. The daemon’s in-process write mutex remains purely a REST-contention reducer, not a correctness dependency.

Role-split topologies

The checklist reads as if every replica were identical, but nothing requires that: each sithbitd listener and the background workers are independent config toggles, so a fleet can split by role — inbound MX edges, an authenticated-submission edge, an IMAP/POP pickup tier, and headless workers — all over one store. The split is the many-instance cloud-store story, so every checklist item applies unchanged; in particular item 1: a role split is a multi-process deployment, and SQLite stays one process per store, full stop. This is a sithbitd story — the standalone smtp-server/imap-server/pop-server binaries remain dev shells, not the production split.

Five toggles produce the roles:

The defaults match the first row ([smtp], [imap], [pop], and [spooler] on; [submission] off), so every preset below writes only the lines that differ — the usual store/TLS/hostname settings from the Configuration reference come on top.

[spooler] enabled = false skips all the background workers as one unit: relay, DSN, the chain pin/send + delete pipeline, the auto-settle sweeper, the reconciler, and the repin migration. Mail is still accepted and spooled — the jobs sit in the shared queues until a worker-enabled sibling drains them. Two things deliberately stay outside the switch: the DMARC RUA/RUF reporting workers keep their own section switches, and the embedded IPFS swarm runs whenever it is configured — DHT participation is the node’s job, not a spooler worker.

# MX edge — accept inbound mail and spool it; a worker sibling drains it.
# [grpc] alone gives this edge RCPT-time postage verification (and
# at-rest sealing key reads) without an [ipfs] provider it never uses —
# the verification-only posture; the pipeline runs on the worker tier.
[grpc]
endpoint = "http://mail-grpc:50051"
[imap]
enabled = false
[pop]
enabled = false
[spooler]
enabled = false

# Submission edge — authenticated client sends only.
[smtp]
enabled = false
[submission]
enabled = true
[imap]
enabled = false
[pop]
enabled = false
[spooler]
enabled = false

# Pickup tier — IMAP + POP readers.
[smtp]
enabled = false
[spooler]
enabled = false
# [imap]
# watch_poll_seconds = 5   # auto-adopted on an IMAP-only instance; see below

# Workers — no listeners, all the background workers (the [spooler]
# default). The chain pipeline runs where the workers run, so the
# [grpc] + [ipfs] sections belong on this instance; the SMTP edges
# carry [grpc] alone, for verification only.
[smtp]
enabled = false
[imap]
enabled = false
[pop]
enabled = false

mail-grpc itself is not a fleet member: it stays a single private-network service the worker tier points at, and a many-instance fleet can safely share one gateway because the store lease serializes each wallet’s chain writes. The reasoning is recorded in the gateway topology design note.

IDLE on a split-out pickup tier

Delivery push is in-process, and nothing delivers on a pickup instance — so its IDLE wakes come only from store polling: the IMAP backend polls the mailbox change counter every watch_poll_seconds and pushes the untagged EXISTS to idlers. Stated plainly:

• New-mail latency is the poll interval, not instant. An idler on a pickup instance learns of a delivery up to watch_poll_seconds after the worker tier lands it.
• Leaving the setting at its 0 default (“trust in-process push”) would leave idlers asleep forever on an instance where nothing delivers, so sithbitd applies a safety rider: IMAP on + both SMTP roles off + watch_poll_seconds = 0 auto-adopts 5 seconds, with an info log saying so. An explicit value is always the operator’s choice, and 0 keeps meaning in-process push whenever an SMTP role is co-resident.
• The known seams above bind with full force here: \Recent is best-effort across instances, and a woken idler re-SELECTs before FETCHing the new message.

Which stores support which split

The store rules are the checklist’s, mapped onto roles. On postgres/aws/azure/cloudflare, any role may run N-wide — queues, leases, and counters are all cross-instance. sqlite and turso allow no split at all — one process per store.

Both split topologies are proven in-tree. An always-on test walks one message across three role instances — real SMTP into an MX edge, chain pin + SendMail on a workers instance, poll-fed IDLE wake then FETCH and POP RETR on a pickup instance — over one SQLite store (mail_spooler’s one_message_crosses_the_role_split_topology; each instance gets its own store handle inside one test process, since real SQLite deployments stay single-process). The same walk runs as a true multi-writer split on postgres, each role booting its own store stack from [store] kind = "postgres", gated on SITHBIT_TEST_POSTGRES_URL (one_message_crosses_the_role_split_topology_on_postgres).

Adding a storage backend

Six backends (SQLite, Postgres, DynamoDB+SQS, Azure Tables/Queues, Turso/libSQL, and Cloudflare D1/Queues/KV/R2) share one behavioral contract, and the plumbing is deliberately small. A new backend touches exactly six places, all but a one-line forwarding entry in mail_store:

1. Cargo.toml — a cargo feature gating the backend’s SDK deps. Declare it in mail_store and keep all complete; each of the five store-consuming binaries (mail-spooler, account-api, ipfs-daemon, ipfs-gateway, mail-console) forwards it in its own [features] table (see Slim-build features);
2. config.rs — a StoreKind variant plus its [store.<kind>] settings struct (never feature-gated: configs parse in every build);
3. a backend module implementing the four repo traits (AccountRepo, MailRepo, JobQueue, KeyedLease) — blobs are orthogonal and stay behind AnyBlobStore;
4. stores.rs — a <Kind>Stores alias with an open constructor (plus its BackendDisabled stub alias), one arm in the with_backend! macro (the workspace’s single backend dispatch point; sithbitd and account-api both route through it), and the enabled/disabled __with_backend_<kind> helper pair;
5. lib.rs exports, feature-gated;
6. tests/mod.rs — an env-gated conformance registration deriving from the canonical test list (skips must be named, with a reason), feature-gated.

The contracts to honor are written where they bind: the counter- allocation rules (atomic, monotonic, gap-tolerant) on the MailRepo trait doc with the three known implementation strategies; the job identity-vs-claim-token split on JobQueue; and the two frozen composite-key codecs in mail_store::keys (pick unit_sep if the store allows control bytes in keys, percent if not — never invent a third). The conformance suite proves all of it against a live instance before the backend ships.

A backend that creates its own cloud resources requests provider-managed encryption at rest when it does so — key configuration is optional, never required. The AWS backend enables server-side encryption on the DynamoDB tables (AWS-owned key) and SSE-SQS on the queues it creates, or a customer-managed KMS key for both when [store.aws] kms_master_key_id is set (see Running a mail server for detail); Azure Storage/Tables and Cosmos are always encrypted at rest by the platform, so no code is needed there.

IPFS: the shared-bucket cluster

The self-hosted IPFS node scales by the same principle as the stores: the bucket is the truth, the nodes are stateless. N embedded nodes (or sithbit-ipfsd daemons) point at one S3/GCS/Azure bucket ([ipfs.blobs] / ipfsd’s [blobs]) and enable [ipfs.cluster] / [cluster] — that’s the whole join procedure: membership heartbeats live in the bucket next to the blocks and pin manifests, so a node needs nothing but the bucket credentials. There is no gossip transport, no bootstrap list, no consensus.

What the cluster coordinates:

• Any-node pin/unpin. Pin manifests are last-write-wins objects in the bucket; every node sees every pin (the reprovide sweep re-reads them), and any node can serve any pinned block over bitswap or GET /ipfs/{cid} — the data has exactly one billed copy, in the bucket.
• Partitioned DHT announces. With [swarm] provide = true, live members split the reprovide keyspace by rendezvous hashing — each root is announced by exactly one member. A member that misses heartbeats for member_ttl_secs is dead; survivors notice at their next heartbeat tick and immediately resweep, taking over its share (remote DHT records carry a ~24 h TTL, so a dead node’s announces stay resolvable while the takeover lands).
• GC. The sweep deletes blocks no manifest references, but only once they are gc_grace_secs old — a pin writes its blocks before its manifest, so in-flight pins are never collected. Sweeps are idempotent; several nodes sweeping concurrently is safe, just redundant.

Failure economics: a dead node costs nothing but its share of DHT announces until a survivor’s next heartbeat tick. Try it: docker compose -f docker-compose.cluster.yml up -d boots two daemons over one minio bucket, and docker/cluster-smoke.sh pins on node 1, kills it, and fetches through node 2.