Runbook: incident response

This runbook is for validator operators. It covers how incidents surface, the triage commands that work today, the two most common failure classes, the emergency stop, what an outage means for your active leases, and what the marketplace can do platform-side.

How incidents surface

Marketplace probe. The marketplace probes every live validator's TEE every 60 seconds with a real SignCertificateVerify call (signed with a short-lived probe JWT). Three consecutive failures, about 3 minutes, flag the validator on the marketplace side. There is no email or pager notification in v1: your operator-facing signal is the dashboard.

Dashboard security strip. The validator dashboard at /validator shows a security strip with five signals:

Trader-visible symptoms. Traders hit your TEE directly over gRPC, so the status codes they report identify the failure class:

Triage checklist

These commands work today. Run them from your deploy checkout of staked-quic-connection-provider unless noted.

1. gRPC health from outside:

   grpcurl -insecure \
     -import-path proto -proto signer.proto \
     <ELASTIC_IP>:443 signer.v1.StakedQuicSigner/Health
   

   -insecure is expected: the leaf certificate is self-signed with the Elastic IP in the SAN. You want provisioned: true and a pubkey_sha256 that matches your identity key.

2. Host logs over SSM (port 22 is closed by design, there is no CloudWatch log group; logs live in journald on the host):

   aws ssm start-session --target <instance-id>
   journalctl -u host-proxy --since "1 hour ago"
   

3. KMS policy drift check (the product's core trust assumption):

   validator-cli security --kms-key-id <key-id-or-alias> --aws-region <region> --json
   

   Exits 1 and prints RED if any Allow on kms:Decrypt in the CMK policy lacks a kms:RecipientAttestation:* condition, or if the CMK is disabled or unreadable. A red result means enclave attestation is no longer enforced on seed decryption: treat it as a security incident, run the emergency stop, and contact security@swqos.dev.

4. EC2 instance health:

   aws ec2 describe-instance-status --instance-ids <instance-id> --region <region>
   

5. Host-proxy metrics (Prometheus, bound to localhost only, so read it from inside an SSM session):

   curl -s http://127.0.0.1:9090/metrics
   

6. End-to-end smoke test (sends 50 transactions through the full pipeline):

   ./deploy/scripts/smoke-test.sh
   

Incident A: enclave unprovisioned after an EC2 reboot

This is the most common incident. A Nitro Enclave boots unprovisioned by design (fail closed): after any reboot, every sign call answers UNAVAILABLE and Health reports provisioned: false until the operator re-injects the KMS-encrypted seed. Nothing auto-provisions.

./deploy/scripts/reprovision.sh --region <region>

The script re-runs the one-time provision binary over SSM using the seed ciphertext persisted on the host (no local keypair needed), then verifies Health. The host-proxy recovers without a restart.

Incident B: wrong audience

Every lease JWT carries an aud claim that the host-proxy checks byte-for-byte against its configured MARKETPLACE_JWT_AUD. If your registered endpoint_url changes without updating the host config (or vice versa), every newly minted lease JWT is rejected with PERMISSION_DENIED "wrong audience". The TEE itself is healthy, so probes that predate the change can mask the problem.

Fix: make the host's MARKETPLACE_JWT_AUD and the marketplace listing's endpoint_url identical again, byte-for-byte, and keep them in lockstep for any future change.

Emergency stop

When you need inbound traffic to stop now (suspected compromise, runaway abuse), validator-cli emergency-stop revokes every ingress rule on the EC2 security group. The block is enforced by the VPC at the hypervisor level and takes effect within seconds.

It is dry-run by default, so an accidental invocation cannot break ingress. The dry run prints the current ingress rules and the planned revoke. Keep that output: it is your record of what to restore.

# Dry run (default): prints current rules, changes nothing
validator-cli emergency-stop --security-group-id sg-0123456789abcdef0 --aws-region <region>

# Apply: revokes ALL ingress rules on the security group
validator-cli emergency-stop --security-group-id sg-0123456789abcdef0 --aws-region <region> --dry-run=false

What it does:

• Calls RevokeSecurityGroupIngress on the security group. Inbound traffic stops within seconds.
• Appends a local audit line (timestamp, operation, params, outcome) to ~/.config/staked-quic/audit/<YYYY-MM-DD>.ndjson, mode 0600. This log is local only; nothing is sent to the marketplace.

What it does not do:

• It does not stop the enclave or the host-proxy: both keep running, and the KMS identity envelope stays sealed.
• It does not delist you. The marketplace only notices indirectly, through probe failures degrading your TEE health. Delisting is a separate admin action (see below).
• It does not freeze leases and it does not refund anyone. If booked epochs will go unserved, revoke the affected leases yourself (next section).

Recovering from an emergency stop. Restoring ingress is manual. The deploy provisions a single ingress rule, tcp/443 from your --allowed-cidr:

aws ec2 authorize-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp --port 443 \
  --cidr <your-allowed-cidr> \
  --region <region>

Alternatively, terraform apply from deploy/terraform/ in the checkout that deployed the host reconciles the security group back to its declared rules.

Lease impact while the TEE is down

The TEE is invoked only at QUIC handshake time. While it is down or unreachable:

• Traders cannot complete new handshakes to your validator. Warm connections already in the SDK pool keep working until they are dropped or evicted, so the outage often shows up as failing reconnects rather than an instant hard stop.
• Marketplace JWT issuance is unaffected (it is a separate service), so traders still receive tokens; they just cannot use them against your endpoint.

There is no automatic compensation. If the outage will eat booked epochs, the honest move is to revoke the affected leases, which books a prorated refund:

• Dashboard: each active lease in the Capacity card has a Revoke button. The dialog requires a reason and previews the exact refund before you confirm.

• API: validator API key auth (the bare 26-character ULID shown once at registration), Idempotency-Key required:

  curl -X POST https://api.swqos.dev/v1/validators/<validator_id>/leases/<lease_id>/revoke \
    -H "Authorization: Bearer <validator_api_key>" \
    -H "Idempotency-Key: 01JXAMP1E0000000000000000Z" \
    -H "Content-Type: application/json" \
    -d '{"reason": "TEE host outage, booked epochs unservable"}'
  

  The response includes refund_amount_lamports and the new status "revoked". Only active leases can be revoked: a pending or completed lease fails with 422 lease-not-active, an already-revoked one with 409 lease-already-revoked.

The refund is floor(remaining_epochs * total_amount / total_epochs), where the current epoch counts as remaining and a lease that has not started yet refunds in full. The amount is booked on the lease row immediately and paid out manually by ops within 24 hours. The trader's outstanding JWTs are not recalled; they simply expire at their TTL (default 600 seconds, so within 10 minutes), and any further token request returns 403 lease-not-active (the token endpoint uses that one slug for every non-active lease, revoked included).

The same dialog is how you cut off a single abusive trader without touching anyone else's lease.

Marketplace-side suspension and takedown

Independently of anything you do on the host, marketplace staff can moderate a listing. Knowing the semantics helps you interpret what traders see:

• POST /v1/admin/validators/{id}/suspend (staff): flips the listing to suspended. Existing leases keep running; only new bookings are blocked. Reversible.
• POST /v1/admin/validators/{id}/takedown (admin only): flips the listing to takedown. Refunding active leases is a separate ops step.
• POST /v1/admin/leases/{id}/revoke (staff): platform-side revocation of a single lease.

Every admin action carries a free-text reason that lands verbatim in the platform audit log. If your listing was suspended and you believe it is in error, contact the marketplace with your validator_id.

Who to contact

For anything security-relevant (suspected key compromise, KMS policy drift, enclave or host compromise, a vulnerability you found), email security@swqos.dev. Include your validator_id, the timeline, the relevant journalctl excerpts, and your local audit lines from ~/.config/staked-quic/audit/. See security reporting for the disclosure process.