Runbook: TEE upgrade

Why upgrades need a runbook

The KMS key policy on your CMK gates kms:Decrypt on the enclave's PCR measurements. A new enclave image (EIF) has new PCR0/1/2 values, so the new enclave cannot unseal the identity seed until the policy is updated to match.

Two things make this simpler than it sounds:

• No key re-wrap happens. The CMK and the seed ciphertext persisted on the host (/etc/staked-quic/seed.bin) stay exactly as they are. Only the PCR conditions in the key policy change, then the new enclave re-provisions from the same ciphertext.
• You never compute PCRs yourself. validator-cli fetch-release writes .pcr-values.json from the K5-signed release attestation. Developers building from source get them from build.sh, which captures the nitro-cli build-enclave output.

The topology is one EC2 instance per validator with one CMK. The upgrade is strictly in place: there is no canary host, no blue/green, no batch rollout. Upgrades are operator-initiated, the marketplace never pushes code to your host.

Prerequisites

• The same repo checkout (and Terraform state) that ran deploy.sh. update.sh exits with an error if there is no state.
• AWS credentials for the deploy IAM user.
• Tools on $PATH: aws, terraform, jq, grpcurl, curl.
• validator-cli installed (cargo install --path validator-cli --locked from the sibling repo root).

Step 1: stage the new release

validator-cli fetch-release --version <X.Y.Z>

This downloads the release from the GitHub Releases base and:

1. Fetches the EIF attestation and verifies its K5 signature. A validly signed attestation for a different version is rejected (the CLI bails on any version mismatch with --version).
2. Downloads enclave.eif, host-proxy, and provision, verifying the sha384 of each against the signed attestation. Files are streamed to a .partial path and only renamed into place after the hash matches, so no failure mode leaves a corrupt artefact at the final path.
3. Writes deploy/artefacts/.pcr-values.json (PCR0/1/2 from the attestation) and .release-version.json (which enables the on-host revocation poller).

Developers iterating on unsigned local builds use ./deploy/scripts/build.sh instead. Local artefacts have no K5 signature, so the update must then run with --dev-no-pollers (test only, it disables the revocation kill-switch poller).

Step 2: run the update

./deploy/scripts/update.sh          # interactive confirmation
./deploy/scripts/update.sh --yes    # skip the prompt

What it does, in order:

1. Validates that PCR0/1/2 are present in deploy/artefacts/.pcr-values.json.
2. Rewrites the enclave_pcr0/1/2 lines in terraform.tfvars. Nothing else changes: region, CIDR, and instance type stay as deployed.
3. Runs terraform plan and refuses any plan that would destroy or replace a resource. If the plan is not purely in-place, the script aborts and tells you to use destroy.sh + deploy.sh instead. Your EIP, TLS keypair, config.toml, and marketplace listing all survive the update.
4. Applies the plan (this updates the KMS key policy PCR conditions).
5. Records the pubkey hash the running host-proxy reports, for the identity continuity check at the end.
6. Over SSM: stops host-proxy and the enclave, downloads the new host-proxy, provision, enclave.eif, and systemd units from the S3 artefacts bucket, restarts the enclave, restarts host-proxy, then re-runs provision against the seed ciphertext already on the host. No local keypair is needed at any point.
7. Verifies the gRPC Health RPC reports provisioned: true and that the pubkey hash is unchanged. An identity mismatch is a hard error: the script exits non-zero and tells you to investigate before serving.

The script is re-entrant. If it fails partway, fix the cause and re-run it.

Downtime, the probe, and your listing

The signing outage covers the enclave stop, artefact swap, enclave boot, and the provision run. The PRD budget for an in-place upgrade is under 60 seconds (US-V06, single-region single-instance v1). During that window the TEE answers UNAVAILABLE on every sign call, so traders with an active lease see failed sign requests until provisioning completes.

How the marketplace reacts:

• The probe calls SignCertificateVerify on every live validator every 60 seconds (MARKETPLACE_PROBE_INTERVAL_SECS, default 60). A probe that lands inside the restart window is recorded as a failed probe_result row and the probe simply tries again on the next tick.
• Your listing stays live. Probe failures never auto-delist a validator. Failed rows lower the 30-day uptime percentage shown on the browse and detail pages and show as a dip in the probe-success sparkline on /validator.
• Three consecutive probe failures (about 3 minutes) trigger a Sev-2 alert that notifies the validator. A normal upgrade costs at most one or two probe ticks and never reaches that threshold.

Pick a quiet moment relative to your booked epochs if you can, but an upgrade within budget is invisible to most traders.

Verify after the upgrade

update.sh already verified provisioned: true and identity continuity. To re-check by hand from the deploy checkout (the proto file ships in the repo):

grpcurl -insecure \
  -import-path proto -proto signer.proto \
  <PUBLIC_IP>:443 signer.v1.StakedQuicSigner/Health

Expect provisioned: true and the same pubkeySha256 as before the upgrade. Then:

• Dashboard (/validator): probe success back to green on the sparkline, and the Security strip shows TEE health healthy and the Cosign verify badge ok (it reads stale if the most recent verification is over an hour old, give it a probe tick or two).

• KMS policy drift check: confirms every kms:Decrypt allowance still carries its attestation condition after the policy rewrite.

  validator-cli security --kms-key-id <key-id-or-alias> --aws-region <region>
  

  Exit code 1 means drift, see incident response.

• End to end (optional): ./deploy/scripts/smoke-test.sh sends 50 transactions through the freshly upgraded TEE.

Rollback

A rollback is just an update to an older release. Stage it, then re-run the same script:

validator-cli fetch-release --version <older-version>
./deploy/scripts/update.sh

Background: the EIF revocation kill switch

Why a release can refuse to boot: revoked-eif-versions.json in the sibling repo is a cumulative list (entries are never removed). When a version is added and merged, a CI job K5-signs the list and republishes it, with no EIF build in the loop. Every host-proxy running a revoked version halts within one 60 second poll. Operators can watch the poller with the host_proxy_revocation_last_success_timestamp_seconds metric on the host's local Prometheus endpoint (127.0.0.1:9090).

What this runbook does not cover

Identity key rotation has no command in v1 and is not part of an upgrade. Changing the identity keypair means destroy, redeploy, re-register, see key rotation. After a plain EC2 reboot (no new release involved) the fix is ./deploy/scripts/reprovision.sh --region <region>, covered in incident response.