Skip to content

Lab acceptance runbook

Use this runbook to qualify a synthetic CertiStack recovery lab before generating a release-evidence campaign. It is deliberately a stop/go process: a running VM, a reachable PBS server, or green unit tests are not acceptance evidence.

Keep customer data, production credentials, source-network addresses, and site-specific VM IDs out of this repository and out of campaign logs.

1. Establish the test boundary

Use synthetic fixtures only. Record the approved fixture population, PVE/PBS major pair, PBS datastore/namespace, controller build, and template/image identifiers in an access-controlled inventory. The approved count must be the published support boundary, not a count inferred from powered-on VMs.

Provision a distinct restore zone/VNet before generating plans. It must have no gateway, SNAT, routed path, or physical uplink. A distinct L2 domain may reuse the source VLAN number and IPv4 range: ARP and IP addresses do not collide across disconnected bridges. When doing so, provide the required isolation attestation and retain it with the campaign. Otherwise use a non-overlapping restore range. Confirm the PVE node can use the intended read-only PBS mapping and each selected fixture has a current snapshot in the approved namespace.

All guests that form one recovered service must attach to the same isolated VNet. The current recovery contract is one flat, air-gapped L2 domain. Guests on it reach each other as far as their own firewalls allow, so a tier that reports on its dependencies can be asserted in tier order. It does not provide routed multi-VLAN topology or application-transaction evidence. A service split across VLANs needs a separate, explicitly designed and qualified topology capability before it is listed as supported; do not represent co-restored per-VM probes as that proof.

2. Accept every template

Before a source appears in an inventory with readiness: ready, retain a record that proves all of the following:

  • Declared and observed OS identity match.
  • Cloud-init or the supported equivalent completed and the expected hostname is present.
  • SSH reaches the fixture when the test requires SSH.
  • The expected QGA virtio channel exists and guest-ping succeeds when QGA is required.
  • The configured workload and probes represent the declared service type.
  • For every HTTPS probe, the service certificate has a SAN matching the declared server_name (or target IP when no name is declared), and its issuing CA is trusted by the PVE node worker. Record the CA fingerprint and trust-store installation with the fixture acceptance record. Self-signed endpoints are acceptable only after their explicit lab CA is trusted; TLS verification bypasses are never an acceptance shortcut.
  • A backup completes and a disposable restore reaches its configured probe stage without touching the source network.

An identity mismatch, missing QGA transport, or unconfigured workload is a template rejection. Keep readiness and workload_configuration blocked until the template is rebuilt and the acceptance record is repeated.

3. Record matrix qualifications

The compatibility audit requires every selected platform and available profile to appear in the accepted inventory. Add a qualification section using only values declared by the compatibility contract:

qualification:
  platform_pairs:
    - { pve_major: 9, pbs_major: 4 }
  storage_profiles: [approved-storage-profile]
  guest_profiles: [approved-firmware-nic-qga-profile]
  backup_profiles: [approved-backup-profile]
  restore_profiles: [approved-isolated-network-profile]

Planned profiles remain visible in the contract but are outside a release support claim until their fixture, automation, and evidence exist.

4. Validate and generate without mutation

Audit the inventory first. A rejection is a stop condition, not a reason to edit a sentinel or lower an expected target count:

go run .lab/audit_compatibility_matrix.go \
  --contract /secure/certistack/compatibility-matrix.yml \
  --targets /secure/certistack/approved-targets.yml \
  --suite release_candidate

Then generate the campaign with a controller-local mode-0600 signing key and the exact approved target count. Validate plans and list cases before contacting PVE:

CERTISTACK_SIGN_KEY=/secure/certistack/controller-signing.ed25519 \
  go run .lab/generate_current_coverage_cases.go \
    --targets /secure/certistack/approved-targets.yml \
    --expected-targets <approved-count> \
    --out /secure/certistack/cases-release \
    --zone-id <restore-zone> \
    --vnet-id <restore-vnet> --vlan-tag <isolated-tag> \
    --ip-range <isolated-cidr> --probe-ip <worker-probe-ip> \
    --allowed-nodes <pve-node>
/secure/certistack/certistack validate /secure/certistack/cases-release
scripts/run-lab-campaign.sh --node <pve-node> \
  --cases /secure/certistack/cases-release --list-cases

If the restore VNet deliberately reuses source addressing, add --isolation-attestation /secure/certistack/restore-isolation.yaml. The YAML must bind the exact source and restore VLAN/CIDR/zone/VNet and affirm all of:

schema_version: 1
source_vlan_id: 101
source_cidr: 192.0.2.0/24
restore_zone_id: csZone01
restore_vnet_id: csVnet01
restore_vlan_id: 101
restore_cidr: 192.0.2.0/24
l2_isolated: true
no_physical_uplink: true
no_gateway: true
no_snat: true
no_routed_path_to_source: true

The generator copies this attestation into the private generated-case directory. CertiStack also checks the preprovisioned VNet for a gateway and SNAT at execution time; the attestation covers the operator-verified L2 and routing boundary that the plan API cannot determine alone.

5. Execute and retain evidence

Run healthy, fault, admission, isolation, interruption/recovery, and upgrade drills separately. A fault drill passes only when exactly its declared VM/probe fails with the expected protocol and description; collateral failures are diagnosis evidence, not a successful fault test. For a release candidate, use --release-evidence with a pinned report signer keyring; it rejects skipped image integrity verification. Validate generated plans before PVE contact; an HTTPS plan that requests insecure_skip_verify or tls_skip_verify is a fixture rejection, not a lab-only exception.

Retain signed reports, verification output, campaign.meta, plans.tsv, cases.tsv, generated plans/manifest, packet capture, interruption journal, node/controller build IDs, and PVE/PBS/image identifiers. A single success report is never evidence for a broader matrix.

See release readiness for the support boundary and release decision rule.