Disaster Recovery and Key Ceremony

Part of the on-prem suite. This runbook applies to the self-hosted Verifiable Compliance Suite. The signing steps use algovoi-key-ceremony, which ships in the bundle. Every attestation it produces verifies offline against your console public key, so your continuity evidence is proven, not asserted.

Objectives

Because the suite runs on your own infrastructure, you set the targets; these are the defaults we design and test against for a single-node deployment with nightly off-site backups.

Objective	Target	Basis
RTO (recovery time)	4 hours single node, 30 minutes with a warm standby	Reinstall the bundle offline, restore the database and archive, restart services
RPO (recovery point)	24 hours default, 1 hour with hourly database dumps	Backup cadence you choose; the archive is write-once, so no in-place data is lost between backups
Key recovery	k-of-n custodians available	Recovery Vault reconstructs a lost key from any k of n shares with no vendor in the loop

RTO and RPO are properties of your backup cadence and standby posture, not of the software. Raise the cadence to lower the RPO; keep a warm standby to lower the RTO.

What to back up

The suite state is a small, well-defined set. Everything below is captured by one signed backup manifest.

Item	Path or setting	Why it matters
Console database	`COMMAND_CENTER_DB` (SQLite file, or PostgreSQL via `pg_dump`)	Signed audit chain, user accounts and two-factor secrets, verified-content index
Records Vault archive	`RV_ARCHIVE_DIR` (default `./records-vault-archive`)	All documents, receipts, access logs, legal holds, redaction tombstones
Console signing key	`COMMAND_CENTER_KEY_FILE` (or the KMS/Vault/PKCS#11 wrapped blob)	Signs the console audit chain; required to continue the chain after recovery
Archive signing key	`RV_ARCHIVE_KEY_FILE` (default `<archive>/archive-key.json`)	Signs every Records Vault evidence stream
At-rest archive key	`RV_ARCHIVE_AES_KEY_B64` (if set)	Decrypts the archive at rest
Recovery Vault file	`VAULT_PATH`	The Shamir-split envelopes protecting your keys

The signing keys are the catastrophic single points. Keep them under a real custody mode (KMS, Vault, or PKCS#11 HSM) or split them with Recovery Vault, and never store the plaintext key beside the database backup. See Bring your own keys.

Back up, then prove the backup

A backup you have not restored is a hope, not a control. Capture the set and sign a manifest that pins every file by SHA-256:

# 1) database (PostgreSQL shown; for SQLite just copy the COMMAND_CENTER_DB file)
docker exec algovoi-postgres-1 pg_dump -U algovoi_control_plane -d algovoi_control_plane -Fc > /backup/console.dump

# 2) sign a manifest over the whole backup set with your console key
algovoi-key-ceremony backup-manifest \
  --item /backup/console.dump \
  --item "$RV_ARCHIVE_DIR" \
  --item "$VAULT_PATH" \
  --label nightly --at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --base / --key "$COMMAND_CENTER_KEY_FILE" --out /backup/manifest

Ship console.dump, the archive, the vault file, and manifest/ off-site. Keep the signing keys on a separate custody path.

Restore and verify

On the recovery host, reinstall the bundle offline (python algovoi_unbundle.py --license @licence.key --out ./algovoi --install bundle.algv), restore the data, then prove byte-parity before you trust it:

# restore the database
pg_restore -U algovoi_control_plane -d algovoi_control_plane --clean /backup/console.dump
# restore the archive + vault file to their original paths, then verify against the signed manifest
algovoi-key-ceremony verify-restore \
  --manifest /backup/manifest/backup-manifest.json --root / \
  --pub-key "$COMMAND_CENTER_KEY_FILE" --sig /backup/manifest/backup-manifest.sig.json

A VERIFY-RESTORE: PASS line means the manifest signature is valid and every restored file matches its recorded digest. The console re-verifies its own audit chain on start; a green posture confirms the chain survived the restore intact.

Key ceremonies

Every sensitive key event should be witnessed and recorded. algovoi-key-ceremony turns each into a Falcon-1024-signed attestation, hash-linked to the previous one, so the whole custody history is one tamper-evident chain a reviewer replays offline.

Genesis

Record the generation of a signing key, naming the operator and witnesses and pinning the new key’s fingerprint (kid).

algovoi-key-ceremony attest --id g1 --type genesis --target console \
  --operator "Alice (CISO)" --witness "Bob (Security)" \
  --at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --key "$COMMAND_CENTER_KEY_FILE" --target-key "$COMMAND_CENTER_KEY_FILE" --out ./ceremony

Split into k-of-n custody

Split the key with Recovery Vault, then attest to the parameters and the custodian labels. The shares themselves are never written into the evidence.

algovoi-key-ceremony attest --id s1 --type split --target console \
  --operator "Alice (CISO)" --shamir "2,3,alice:bob:carol" \
  --witness "Bob (Security)" --witness "Carol (Legal)" \
  --at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --key "$COMMAND_CENTER_KEY_FILE" --prev ./ceremony/g1.attestation.json --out ./ceremony

Rotation

On schedule or after a suspected exposure, generate a fresh key. Retain the old public key so historical signatures still verify; new writes use the new key. Attest with --type rotation.

Recovery drill

Periodically prove the split works: reconstruct the key from k shares in a clean room, confirm the recovered fingerprint matches, then record the drill and its outcome with --type recovery_drill. This is the evidence an auditor asks for.

Disposal

When old key material is securely destroyed, record it with --type disposal so the lifecycle is closed out on the chain.

Verify the whole custody chain at any time, offline:

algovoi-key-ceremony verify --pub-key "$COMMAND_CENTER_KEY_FILE" --chain ./ceremony
# CHAIN: PASS (N records) — every record signed, every link intact

Failure-mode runbook

Scenario	Action	Target
Node lost	Reinstall bundle offline, restore last backup, `verify-restore`, restart services	Within RTO
Database corrupted	Restore the last good dump, let the console re-verify its audit chain	1 hour
Console or archive key lost	Reconstruct from Recovery Vault shares (k custodians), attest a `recovery_drill`, resume	k custodians available
KMS or Vault access lost	Fail over to a second custody endpoint, or recover the key from Recovery Vault shares	Within RTO
A custodian leaves	`resplit` in Recovery Vault to revoke the old shares and issue new ones, attest the change	Same day
Suspected key exposure	Rotate the signing key, attest `rotation`, then `disposal` of the old material once retired	Same day

Verify everything offline

Nothing in this runbook needs a network connection or a call to AlgoVoi. Backups verify against a manifest you signed, the audit chain re-verifies on start, and the custody chain replays under your own public key. Recovery Vault reconstructs keys from your custodians’ shares alone. That is the whole point: continuity you can prove to a regulator without depending on a vendor being reachable.

Get started

Platform

Concepts

Keystone

Agent Mesh

Substrate 2

Commercial

Open substrate

Payment protocols

Chains

Integrations

Help

Company

Disaster Recovery and Key Ceremony

Objectives

What to back up

Back up, then prove the backup

Restore and verify

Key ceremonies

Failure-mode runbook

Verify everything offline

​Objectives

​What to back up

​Back up, then prove the backup

​Restore and verify

​Key ceremonies

​Failure-mode runbook

​Verify everything offline

Objectives

What to back up

Back up, then prove the backup

Restore and verify

Key ceremonies

Failure-mode runbook

Verify everything offline