Monitoring and backups

What to back up

The store — the store volume in the compose files, or wherever [store] points — is the only state that matters, and its pieces have very different values:

Two SQLite copy rules: a live database is only complete with its -wal and -shm sidecar files, and the distroless images have no shell — so from a container, docker cp all three files out (a copy missing the WAL reads as empty or stale). For a consistent snapshot prefer stopping the service first, or run sqlite3 sithbit.db ".backup ..." from the host against a mounted volume.

Cloud stores (kind = "aws" / "azure", or "postgres" on a managed instance like Cloud SQL) move this problem to the provider: durability comes from DynamoDB/S3/Azure Storage/the managed database (with GCS behind the s3 blob kind on Google Cloud), and only the key files above still need your own backups.

Logs

Every binary — sithbitd, account-api, domain-sithbit, and mail-grpc — logs structured tracing lines to stdout — docker logs <service> in the compose stacks. The default level is info; filter with RUST_LOG using target=level directives:

RUST_LOG=info,mail_spooler=debug,sqlx=warn

The same RUST_LOG filter also shapes what the OTLP export (below) sends — it sits in front of both the console and the exporter.

Telemetry export (OTLP)

Every binary can push traces and metrics to an OpenTelemetry collector over OTLP/gRPC. Export is off by default and enabled per service by the presence of the [observability.otlp] config section (defaults shown commented in every example config):

[observability.otlp]
# endpoint = "http://127.0.0.1:4317"
# metrics_interval_seconds = 60

mail-grpc is env-configured like the rest of its settings: OTLP_ENDPOINT (absent or empty = no export) and OTLP_METRICS_INTERVAL_SECONDS.

The design is push only — no Prometheus scrape endpoint. Prometheus users run an OTel collector with a Prometheus exporter and point the services at it. For a working dev example, the docker-compose.otel.yml overlay boots a collector with the debug exporter and turns every service’s export on:

docker compose -f docker-compose.yml -f docker-compose.otel.yml up -d
docker compose logs -f otel-collector    # spans + metrics print here

Two gotchas:

• The standard OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / …_METRICS_ENDPOINT env vars override the configured endpoint inside the exporter — leave them unset for the config file to be authoritative.
• Traces ride the RUST_LOG filter: a target silenced for logging is also not exported.

The metrics (all under the sithbit. prefix, labeled as noted):

The job queues

All background work rides durable job queues: chain (encrypt → IPFS pin → on-chain SendMail), relay (outbound SMTP), chain_delete (expunge teardown), dsn (DSN status notifications), and — only while an [ipfs.repin] migration is configured — repin. Jobs are at-least-once with a visibility timeout; failures retry with backoff, and a job that keeps failing is buried to a dead-letter queue with a reason. The two numbers worth watching are depth (backlog) and age of the oldest job (a stuck consumer).

SQLite — the jobs and dead_jobs tables:

-- depth and oldest job per queue (timestamps are unix seconds)
SELECT queue, COUNT(*), MIN(created_at) FROM jobs GROUP BY queue;
-- poison jobs, with why they died
SELECT queue, reason, died_at, payload FROM dead_jobs;

AWS — SQS queues named <queue_prefix>-chain, -relay, -chain-delete, -dsn, and -dead; watch the standard ApproximateNumberOfMessagesVisible / ApproximateAgeOfOldestMessage CloudWatch metrics.

Azure — Storage queues under the same <queue_prefix>-* names, with -dead for buried jobs; watch approximate message counts.

A buried job’s payload is self-describing JSON (a type field plus the job’s parameters). The interactive way to inspect and re-drive dead jobs is the sithbit-console TUI (its Queues tab lists depths and dead letters with confirmed requeue/discard keys — see the console tutorial); underneath it is an account-api admin call (a wallet on its admin_wallets allowlist): GET /v1/admin/dead-jobs lists buried jobs with their queue, reason, and payload, and POST /v1/admin/dead-jobs/requeue (echo a listed entry back) fixes it onto its source queue with attempts reset — POST /v1/admin/dead-jobs/discard deletes it instead. Queue depths are GET /v1/admin/queues. On the cloud backends a listing claims each returned entry for five minutes (the id is a claim token, like a worker’s receipt handle), so requeue/discard within that window; a lapsed entry simply lists again later. Without the admin API, the manual fallback still works: fix the cause and re-insert the payload (SQLite: copy the row back into jobs with attempts = 0, visible_at = now; SQS/Azure: send the body to the source queue).

Buried jobs do not pile up forever: the worker role runs an hourly prune that discards dead-letter entries buried longer ago than [spooler] dead_retention_days (default 30; 0 disables the prune entirely — see Configuration). Every backend stamps the bury time into the dead message itself, so entries age correctly across restarts; a residual entry with no readable date (e.g. one buried by an older build) counts as older than any cutoff and is pruned, not spared. Requeue or discard a poison job you care about within the retention window.

The console also carries a balances pane (press b on a wallet): its native SOL balance, its mailbox’s default stamp price and received mail count, and the prepaid stamps each other loaded wallet holds toward it. Unlike every other pane, this one reads chain state directly — the mailbox/frombox figures come gRPC-direct from mail-grpc’s GetMailbox/GetFrombox, and the SOL balance from a Solana JSON-RPC getBalance — not through the account API. It is a read-only spot check; its two endpoints (gateway_endpoint, rpc_url) are configured alongside the console’s api_url (see Configuration).

Outbound quotas and suspension

Every authenticated sender carries rolling hour/day counters of the external (relayed foreign-domain) recipients it has been accepted for — local, on-chain-stamped mail never counts — enforced against the age-ramped allowances configured in [smtp.quota] / [submission.quota] and the account API’s twin [quota]. Alongside them rides a per-account suspend flag, honored by every server whether or not quotas are enabled.

Both are administered over the account API (a wallet on its admin_wallets allowlist, like every /v1/admin route):

• GET /v1/admin/accounts/{wallet}/quota — the wallet’s rolling usage and the allowances actually in force at its current age: {last_hour, last_day, suspended, account_age_weeks, quota_enabled, hourly_allowance, daily_allowance}. 404s for a wallet with no account row (i.e. one that never logged in — counters alone don’t create an account).
• PUT /v1/admin/accounts/{wallet}/suspend with body {"suspended": true} (or false to lift it) — 204 on success, 404 for a missing account.

What a refused sender sees on the wire, per surface — the reference for debugging “why can’t this account send/collect”:

The account-disabled replies are deliberately distinguishable from a bad-credentials refusal, and deliberately safe to disclose: every one of them is issued only after the presented credentials verified, so only the account holder ever learns the account is suspended — a stranger probing passwords still sees the ordinary bad-credentials refusal.

Quota refusals surface in telemetry as sithbit.rcpt.refusals{reason="custom_452"} — a growing count means senders are hitting their allowances, which is the control working, not an outage.

Complaint handling is deliberately manual for now: on an abuse report, suspend the wallet via the admin API above and lift the flag once resolved — suspension stops SMTP, IMAP, POP, and compose in one switch. Automated ARF (abuse-report) ingestion is deliberately deferred.

Chain states

Every delivered message copy tracks its progress to the chain in messages.chain_state:

SELECT chain_state, COUNT(*) FROM messages GROUP BY chain_state;

A copy sitting in a non-terminal state (received, pinned) for more than 15 minutes is picked up by the reconciler, which sweeps every 5 minutes and re-enqueues a chain job for it — duplicates are harmless by design. A growing received count on a chain-enabled deployment therefore means the pipeline itself is unhealthy: check the chain queue depth, the dead-letter queue, and connectivity to mail-grpc and IPFS.

Liveness

Every binary serves two HTTP health endpoints on a loopback health listener (the [health] section in every binary’s TOML config; enabled = false disables). Default ports, one per binary so a dev host can run them all:

• GET /healthz — 200 while the process serves (liveness).
• GET /readyz — 200 once startup finished (listeners bound); 503 lists what’s still waiting (readiness).

Because the runtime images are distroless (no shell, no curl), every binary also takes a --health-probe flag: it loads the same config, GETs its own /readyz, and exits 0/1. The compose files use exactly that as their healthcheck: (see docker-compose.yml), and docker/smoke.sh waits for the services to report healthy. One caveat: mail-grpc runs host networking in the chain profile, so its probe port 8193 lives on the host — move it with [health] bind_addr (or MAIL_GRPC_HEALTH__BIND_ADDR) if something else holds it.

Also useful:

• SMTP/IMAP/POP answer with a protocol banner on connect — docker/smoke.sh scripts exactly this for the dev stack.
• mail-grpc’s ListAliases RPC answers UNAVAILABLE (“still backfilling”) until the alias index has completed its first sync — a finer-grained readiness signal for the index than /readyz, which only tracks the gRPC listener (the sithbit.alias_index.staleness_seconds metric covers ongoing sync health).