Postmaster key custody

Since the delegation cutover the postmaster is not a key at all — it is a hidden set of keys committed by a 32-byte Merkle root on the postoffice account, produced in an offline key ceremony. This page is the operator runbook for that ceremony: what to generate, what to store where, how the generation stepping works, which of the two owner-run commands to reach for and when, and — honestly — what today’s tooling can and cannot recover when a seed is lost.

The other half of the admin surface, the standing delegate, is an ordinary hot wallet with strictly operational powers. The owner rotates it with postmaster delegate; the ceremony seeds behind the commitment root are rotated wholesale with postmaster commitment. Those two commands run on completely different clocks, and getting that cadence right is most of the postmaster’s job — see When to run which command. The command shapes themselves are on The Postmaster.

CLI build note: the entire sithbit postmaster command tree used throughout this runbook (init, commitment, withdraw, delegate, set-*-fee, ksk get/set/iana, and the read-only get) is gated behind the CLI’s postmaster feature, which is not in the default feature set. Build an admin CLI with cargo build -p mail-client --features postmaster (or --all-features). The default build exposes only the read-only query, as sithbit postoffice postmaster.

The two owner-run commands at a glance

The owner never signs day-to-day admin — that is the delegate’s job. What the owner does sign are the ownership operations, and in normal running only two of them recur: delegate and commitment. They look similar on the command line (both present the ceremony seeds and spend one committed key), but they answer opposite questions and run on opposite schedules.

Both are ownership operations, so both share the rotate-on-use mechanics below: each spends one revealed generation and installs the next generation’s root in the same instruction. Running either one advances the generation the other will prove against next time — the CLI recovers the current generation from the chain, so you never track this by hand.

What the ceremony produces

The ceremony input is N ≥ 2 seed files — ordinary Solana keypair JSON files whose 32-byte secret halves act as independent master secrets (the public halves are ignored; any 32-byte secret in that container works). From each seed a deterministic chain of signing keys is derived by hashing; the derived keys’ public hashes from every seed are interleaved into one Merkle tree; and the tree’s root is the commitment root the postoffice stores at postmaster init.

The picture below is the whole custody scheme in one frame — where each seed and leaf sits under the single on-chain root, and how one signed op reveals a leaf and rotates the root:

Nothing about the member keys is published — the chain holds only the root. An ownership operation (commitment, withdraw, delegate) reveals one derived key: the key signs the transaction and presents its Merkle membership path, which the program verifies against the stored root.

Rotate-on-use, in generations. Every ownership operation spends the whole revealed generation and installs the next generation’s root in the same instruction. Generation g’s tree is derived from each seed’s key window g·K … g·K+K−1 (K = --keys-per-seed), so the next generation is a fresh, disjoint key set from the same seeds — no new ceremony needed. A key that has appeared on-chain is never accepted again.

This is why routine operation never needs commitment: the seed set replenishes itself, generation after generation, out of the same seeds. You reach for commitment only when the seed set itself must change.

The CLI is stateless. There is no counter file: the current generation is recovered by re-deriving candidate trees from the seeds and matching their roots against the chain (bounded scan). Every ownership command takes the same arguments — the seed files in ceremony order, plus the ceremony parameters:

sithbit postmaster withdraw \
  --seed vault-a/seed0.json --seed vault-b/seed1.json \
  --keys-per-seed 2 --key-index 0 -k payer.json

What to store where

Two rules produce the whole custody scheme:

1. Each seed lives in its own vault, held by a different person, organization, or physical location. The seeds are the ownership; anyone holding all of them owns the postoffice outright. Splitting them is what makes the postmaster resistant to any single theft, phish, or subpoena — an attacker with one vault has nothing usable on its own.
2. The ceremony parameters are recorded with BOTH (all) seeds: the seed count, the seed order, and --keys-per-seed. These are not secret, but every ownership operation must present exactly the values postmaster init used — a ceremony rebuilt with the wrong parameters derives a different tree and its proofs are refused on-chain (error 67, InvalidCommitmentProof). Write them on paper in every vault.

The fee-paying -k keypair on ownership commands needs no admin standing — any funded wallet can pay; it is not custody material.

When to run which command

This is the section to internalize. The two owner commands are not interchangeable and they do not run together; keeping their cadences straight is what a healthy deployment looks like over its lifetime.

postmaster delegate — routine, scheduled hot-key hygiene

The standing delegate is a hot key by design (see Rotating the delegate for what it can and cannot do). Because it is hot, it is rotated often and on a calendar — quarterly is a reasonable default — plus immediately on any suspicion the delegate host was compromised or the person holding it left. Each rotation:

sithbit postmaster delegate <new_wallet> \
  --seed vault-a/seed0.json --seed vault-b/seed1.json -k payer.json

repoints the postoffice’s delegate_address at a new wallet and, being an ownership operation, steps the ceremony generation as a side effect. In practice these scheduled delegate rotations are the main reason the owner takes the seeds out of their vaults at all: a short signing session, then the seeds go back. Nothing about the owner set changes — same seeds, same custodians, next generation.

postmaster commitment — rare, event-driven ownership change

commitment installs a successor commitment root: a brand-new ceremony’s root, replacing the entire hidden owner set. It changes who owns the deployment, so it runs only when ownership genuinely moves, not on any clock:

sithbit postmaster commitment \
  --seed vault-a/seed0.json --seed vault-b/seed1.json \
  --keypair payer.json

Run it when:

• Adopting the ceremony world. Migrating a deployment off the old single-key postmaster is a one-time install of the first commitment root — the old owner signs it, and from then on only the ceremony seeds prove ownership.
• Handing ownership over. The new owners run their own offline ceremony on their own seeds and hand you the resulting root; you install it. Afterwards only their seeds can sign, and your old seeds are out of the trust set forever.
• Retiring a seed from the set. If a vault is at risk — a custodian is leaving, a location is no longer trusted — you run a fresh ceremony on a new mix of seeds and install its root while you still hold all the current seeds. This is the proactive move the recovery story below hinges on: rotate a seed out before it is gone, not after.

The on-chain effect is narrow: commitment writes only the new commitment_root and leaves delegate_address untouched — the operating key keeps working straight across an ownership handover. (delegate, by contrast, writes the new delegate_address and steps the root; the two never overlap.)

How the two clocks relate

Many delegate rotations happen between any two commitment installs. A delegate rotation is a small, routine act against an unchanged owner set; a commitment install is a larger, deliberate event that replaces that set and needs a fresh offline ceremony first. If you find yourself reaching for commitment on a schedule, that is a smell — routine hygiene is delegate’s job, and the generation stepping already refreshes the committed keys under you for free.

Rotating the delegate

The standing delegate is the opposite of the ceremony seeds: a hot key by design, and safe to be one. Its powers are operational only — authorize/deactivate domains, tune capped fees, publish the root KSK, waive bulk-alias fees. It can never sweep postoffice funds, touch the commitment root, or change what the ownership keys are. The worst a stolen delegate does is operational damage (bounded further by the deactivation timelock), and a single ownership operation revokes it.

Recommended cadence:

• Rotate on a schedule — quarterly is a reasonable default; the cost is one sithbit postmaster delegate <new_wallet> --seed … --seed … (an ownership operation, so it also steps the ceremony generation).
• Rotate immediately on any suspicion of host compromise, and on personnel changes that touched the delegate host.
• domain-sithbit needs no redeploy: it re-reads its delegate_key_file from the configured key source — disk or Azure Key Vault — on every POST /domain, so after rotating on-chain, swap the key file (or vault secret) in place and the next request signs with the new key. No restart, no signal, no downtime.

Keep the delegate keypair on the host that needs it and nowhere else; back up nothing — a lost delegate key is a delegate away from irrelevance, which is precisely the point of the split.

Losing a seed: the honest recovery story

Read this section before you rely on the N-seed split. The design goal is that losing one seed must not lose the postoffice, and the chain upholds it — but today’s CLI does not yet, and this runbook will not pretend otherwise.

What the chain requires from a recovery is exactly one valid ownership operation: a signature from any single committed key of the current generation plus its membership path. The successor root installed by that operation is opaque to the program — so a survivor who can produce one proof can install a brand-new ceremony’s root (fresh seeds, held by new custodians) with postmaster commitment and the lost seed is out of the trust set forever. One proof is full recovery.

Producing that one proof without the lost seed is the catch. The Merkle path runs through sibling hashes derived from every seed’s secret — the survivor can re-derive their own keys, but the lost seed’s contribution to the current generation’s tree must come from somewhere. It is public material (leaf hashes, not keys), and the ceremony library can rebuild the tree from one secret plus the other seeds’ published leaf hashes — but only if those hashes were exported while the seed was still available, and they are per-generation: the genesis leaf list is useless once one ownership operation has stepped the tree to generation 1.

Today’s limitations, plainly:

• The CLI cannot run a one-seed recovery. Every ownership command requires every --seed file and re-derives all leaves from the secrets. There is no flag to substitute a lost seed’s public leaf hashes, and no export-leaves command to produce the artifact in the first place. The library layer (mail_client’s ceremony module) supports the rebuild and it is covered by tests — but exercising it today means writing code against that library, not running a shipped command.
• Consequence for operators now: treat the seed set as all-or-nothing until the survivor tooling ships. Losing any seed means ownership operations stop working, and the practical mitigation is to run commitment to a fresh ceremony while you still hold the remaining seeds and the failing one — i.e. rotate out a seed at the first sign its vault is at risk, not after it is gone.
• If you want to be ready for the survivor flow anyway: after init and after every ownership operation, export the new current generation’s public leaf hashes and store a copy in every vault (they are not secret). Whoever runs an ownership operation holds all seeds at that moment, so the export costs nothing. When the survivor tooling lands, those artifacts — plus any one seed — are exactly what it will consume.

See also

• The Postmaster — the delegate/postmaster power split and every command’s shape.
• Trust assumptions and threat model — where the postmaster and delegate sit in the trust boundaries.
• domain-sithbit: domain verification — the self-service flow the hot delegate key serves.