Runbook: key rotation

Scope: what rotates today

There is no rotate-identity command. The full validator-cli surface is fetch-release, register, set-payout-wallet, security, and emergency-stop. This table is the honest state of rotation in v1:

Marketplace JWT signing key (kid rotation)

The marketplace signs every lease JWT with an ECDSA P-256 key held in AWS KMS (alg: ES256). The JWT header carries a kid, and your host-proxy verifies incoming JWTs against a static kid -> pubkey trust set in /etc/staked-quic/config.toml. The trust set normally has one entry and two during a rotation overlap. The design budgets a 24-hour overlap window during which both the current and previous marketplace pubkeys verify.

The rotation is coordinated by the marketplace:

1. The marketplace generates the new KMS key and publishes the new (kid, pubkey) pair to operators.

2. Your action: append a second [[jwt.keys]] entry on the host (reachable over SSM) and restart host-proxy. Downtime is seconds, and both kids verify during the overlap.

   # /etc/staked-quic/config.toml
   [[jwt.keys]]
   kid = "mkt-jwt-v1-2026q2"
   pubkey_pem = """
   -----BEGIN PUBLIC KEY-----
   ...new marketplace P-256 public key...
   -----END PUBLIC KEY-----
   """
   

3. The marketplace flips its signing kid. New trader JWTs (default TTL 600 seconds) carry the new header, old in-flight JWTs still verify against the old entry.

4. After the overlap you remove the old [[jwt.keys]] entry at leisure and restart host-proxy again.

Traders do nothing: their SDK refreshes JWTs automatically and the new tokens just carry the new kid.

Payout wallet rotation

The only true self-service rotation in the product. Use it to move earnings to a cold wallet or split custody:

validator-cli set-payout-wallet \
  --identity-keypair /path/to/validator-keypair.json \
  --payout-wallet <BASE58_ADDRESS> \
  --dry-run   # remove to apply

The CLI signs a dedicated set-payout challenge with your identity keypair file (never through the enclave) and calls PUT /v1/validators/payout-wallet.

• Applies to bookings made after the update only. The payout wallet is frozen onto each lease at booking time, so already-booked leases keep paying the old address.
• The payout wallet card on the dashboard shows the current address and a prefilled copy of this command.

Validator identity key: there is no rotation command

The identity seed exists in exactly two places: inside the running enclave and as a KMS-encrypted ciphertext on the EC2 host. There is no supported way to swap it in place, and re-keying your Solana validator identity on-chain is an operator action outside the marketplace entirely.

The honest marketplace-side procedure for both scenarios below is destroy + redeploy + re-register, because deploy.sh is what KMS-encrypts the seed and provisions it into the enclave. There is no shortcut.

First, cut traffic with emergency-stop

For a suspected compromise, stop inbound traffic before anything else:

# Preview (the DEFAULT is dry-run): prints current ingress rules
# and the planned revoke, changes nothing
validator-cli emergency-stop --security-group-id sg-0123abcd --aws-region us-east-1

# Actually cut traffic
validator-cli emergency-stop --security-group-id sg-0123abcd --aws-region us-east-1 --dry-run=false

What it does: revokes every ingress rule on the EC2 security group, so inbound traffic stops within seconds at the hypervisor level. The enclave keeps running and the KMS envelope stays sealed. A local NDJSON audit line is appended under $XDG_CONFIG_HOME/staked-quic/audit/.

What it does not do: it never talks to the marketplace. It does not delist you, does not freeze or revoke leases, and refunds nothing. Restoring ingress afterwards is manual (re-add the security group rules yourself). See Incident response for the full triage flow.

Scenario A: identity key compromise (new keypair)

1. Cut traffic with emergency-stop --dry-run=false as above.

2. Revoke active leases from the dashboard: each revocation requires a reason and books a prorated refund of floor(remaining_epochs * total / total_epochs) for the trader. Their JWTs die at expiry, within 10 minutes.

3. Tear down the deployment from the sibling repo checkout:

   ./deploy/scripts/destroy.sh
   

   This destroys the EC2 instance, EIP, KMS key, S3 bucket and VPC, and wipes local secrets. It preserves artefacts/ and trader-keypair.json, so the redeploy is fast.

4. Re-key your validator identity on-chain with your normal Solana operator tooling. This step is outside the marketplace.

5. Redeploy with the new keypair (the MARKETPLACE_JWT_PUBKEY_PEM, MARKETPLACE_JWT_AUD and MARKETPLACE_JWT_VAL_PK environment variables are required, with MARKETPLACE_JWT_VAL_PK set to the new pubkey):

   ./deploy/scripts/deploy.sh \
     --identity-keypair /path/to/NEW-validator-keypair.json \
     --region us-east-1
   

   deploy.sh KMS-encrypts the new seed under a fresh CMK, provisions the enclave over SSM, and verifies gRPC Health reports provisioned=true with the matching pubkey.

6. Register the new identity. A new pubkey means a new listing and a new validator_id:

   validator-cli register \
     --identity-keypair /path/to/NEW-validator-keypair.json \
     --endpoint-url https://<NEW_EIP>:443 \
     --price 2.5 \
     --bio "Your listing bio" \
     --region us-east-1
   

   The new API key is written once to <validator_id>.api-key (mode 0600) next to the keypair and never shown again.

7. The old listing stays tied to the compromised pubkey and you cannot delete it yourself. Contact the marketplace so an admin suspends it.

Scenario B: KMS key loss (same keypair)

If the per-validator CMK is deleted, disabled, or its attestation policy is broken, the seed ciphertext on the host can never be decrypted again and reprovision.sh cannot recover. Confirm the diagnosis first:

validator-cli security --kms-key-id <key-id-or-alias> --aws-region us-east-1 --json

This exits 1 if the CMK is disabled or unreadable, or if any Allow on kms:Decrypt lacks the kms:RecipientAttestation condition. The check runs locally and never reports to the marketplace. The KMS drift badge on the dashboard security strip is a v1 placeholder hardcoded to "In sync" and does not detect CMK policy drift, so trust the CLI exit code, not the badge.

Recovery, keeping the same identity keypair:

1. ./deploy/scripts/destroy.sh
2. ./deploy/scripts/deploy.sh --identity-keypair /path/to/validator-keypair.json --region <region> with the same MARKETPLACE_JWT_* environment variables as the original install.
3. Do not re-register. POST /v1/validators returns 409 validator-already-registered for a pubkey that already has a listing. Your listing, history and ratings survive.
4. The redeploy allocates a new EIP. If your registered endpoint_url embeds the old IP, update it from the dashboard listing editor, and make sure MARKETPLACE_JWT_AUD on the host equals the registered endpoint_url byte for byte (the __PUBLIC_IP__ placeholder in MARKETPLACE_JWT_AUD resolves to the new EIP automatically during deploy). A mismatch rejects every lease JWT with PERMISSION_DENIED: wrong audience.

What is NOT automated

Be explicit with yourself about the manual surface:

• There is no scheduled or on-demand identity key rotation. The only path is destroy + redeploy + re-register.
• Marketplace JWT key rotation requires a config edit and a host-proxy restart on every validator host. Nothing pushes the new key to you.
• emergency-stop never delists you, never freezes leases, never refunds. Re-opening ingress afterwards is manual.
• Lease revocation is per-lease and manual, from the dashboard or POST /v1/validators/{id}/leases/{lease_id}/revoke.
• After an EC2 reboot the enclave boots unprovisioned by design. Run ./deploy/scripts/reprovision.sh --region <region> yourself, nothing auto-provisions. See Incident response.
• A TEE upgrade does not re-wrap the seed. update.sh only changes the PCR conditions on the same CMK and re-provisions from the same on-host ciphertext. See TEE upgrade.

Related runbooks