Plan reference

A plan is a YAML file that starts with version: "v1". Section 1 shows its layout. Section 2 explains how the main settings behave. Section 3 defines the probes. Section 4 explains why a plan is a literal document. Section 5 lists every key the parser accepts, treats as an alias, or rejects, and is the only per-key table. For complete plans to copy, see Example plans.

1. Plan layout

Beside its identity (version, plan_id, name), a plan has these blocks:

compliance: the frameworks and retention the report records, and the Enterprise Edition's signing key.
admission_control: how busy the node may be before a run starts.
network: the isolated sandbox network.
storage: the overlay storage, and optional PBS defaults.
tiers: the VMs to restore in boot order, each with its source backup and its probes.

examples/minimal-single-vm.yaml is the smallest useful plan. It restores one VM and checks that its SSH listener answers; the other example plans are on the example plans page.

With use_backup_config: true, CertiStack reads the disk layout and the safe compute and firmware settings from the snapshot (see "Backup configuration inheritance"). A VM in network_recovery.mode: preserve keeps its backup's net0 MAC and model whether or not use_backup_config is set, and a plan that sets a mac or nic_model other than its backup's is refused (see "NIC MAC addresses").

Editor completion and checks

An editor with a YAML language server (VS Code with the Red Hat YAML extension, Neovim, JetBrains IDEs) can complete a plan's keys, offer the values a key accepts (probe types, network.mode, network_recovery.mode, compliance frameworks), show what each key means and mark a mistake as you type it. certistack schema prints the JSON Schema it reads. Save it next to the plan and make this the plan's first line:

# yaml-language-server: $schema=./certistack-plan.schema.json

certistack schema > certistack-plan.schema.json

The schema is generated from the types the loader reads, so it offers exactly the keys section 5 lists and matches the binary that printed it; regenerate it when you upgrade. It marks the keys that are aliases or rejected as deprecated, states the ranges and the keys each probe type reads, and flags a key the loader would refuse. It does not replace certistack validate, which also checks what spans keys (an address inside network.ip_range, a MAC used once, a snapshot of the VM's own backup group).

2. Configuration field notes

Section 5 gives each key's type, range and status. This section explains what the settings do.

`compliance`

Audit metadata, and the key the Enterprise Edition signs with.

frameworks must name at least one label, because an empty list cannot be told from an omission. certistack init writes the placeholder CHANGE, which is refused until you replace it. Every label in the section 5 list is accepted and printed exactly as written. The article codes are DORA-Article-12, SOC2-Type2-CC7.4, HIPAA-164.308-A7 and ISO27001-A8.13; CIS_V8 and NONE have no longer form. A framework label is not a certification.
retention_days is the retention the evidence is held under (2555 for seven years). A missing key or 0 is refused: the report would record no retention.
sign_key_path is the private key file the Enterprise Edition's certistackd service signs its reports with. It is optional, and the Community Edition ignores it: that edition writes its reports unsigned, never reads, checks or requires the file, and accepts the field only so that one plan file works under both editions. validate and plan print one line when a plan sets it, saying so. The path is never sent to the node worker.

`admission_control`

Protects production workloads on the hypervisor host. A run starts only while the node is under both limits. Otherwise it waits defer_retry_interval_sec and checks again, up to max_defer_retries times, and then fails before it creates anything. The four limits are required: a plan that omits one fails validation. The examples use 85.0, 12.0, 30 and 10.

Memory. Before creating any sandbox resource, CertiStack reserves the declared maximum memory of every VM in the plan (each verified tier stays running until the end, so the last tier boots while all earlier ones are still resident) plus 256 MiB per sandbox VM for host overhead, adds that to current host usage, and holds the total to max_host_ram_percent. Ballooning is not counted as guaranteed free capacity. A plan that would overcommit the node fails before PBS mapping.
I/O wait. The share of the node's CPU time spent waiting for I/O over a half-second interval, read from /proc/stat, with guest running time counted once. An interval that is not a measurement (no CPU time passed, or a counter went backwards) is sampled again, up to three times. If none gives a measurement, admission treats the I/O wait as unknown and defers, as for a limit exceeded. After max_defer_retries the run fails before anything is created, naming the unmeasurable I/O wait. Unknown load never counts as idle.
Finite numbers. Limits must be finite (.nan and .inf are refused when the plan loads), and every duration in seconds (defer_retry_interval_sec, startup_grace_period_sec, boot_timeout_sec, soak_time_sec, a probe's timeout_sec) is at most 604800 (7 days).

Limiting concurrent boots. A tier starts its VMs one after another, so their first minutes of boot overlap, when each guest writes most to its new overlays. On a node whose overlay storage shares a disk with the cluster file system's database (the run warns about it), overlapping boots can delay corosync and cost the node its cluster membership. max_concurrent_boots: 1 or 2 spaces them out, across tiers too: before it starts a VM, the run waits while that many VMs started within the last boot_settle_sec (default 180). The wait is logged, counted in the run's admission time, and shown by certistack plan. The integrity scan already spaces starts by about one scan, so the limit matters mostly for VMs with small disks, and in test mode (--skip-integrity, which needs CERTISTACK_ALLOW_TEST_MODE=1), which has no scan.

`network`

The isolated Proxmox VE Software-Defined Network (SDN) sandbox. The zone and VNet IDs may stay the same across scheduled runs. With the default lifecycle: ephemeral, a successful run removes what it created. Before it creates anything, CertiStack reads the exact zone and VNet IDs. If either already exists, it fails closed without reusing, changing or deleting it, so a plan can never adopt a customer's network or an interrupted sandbox. Review the PVE resource or recover the interrupted run, then retry (Issue 10).

The SDN lock. Applying SDN in PVE is all or nothing, on every node. CertiStack creates and removes its zone and VNet under PVE's SDN configuration lock and applies only its own change. PVE refuses the lock while the cluster has SDN changes nobody applied, and the run is then refused before it creates anything; plan reports it too (Issue 12). The lock needs SDN.Allocate on /sdn, which the documented role grants. CertiStack journals the lock request, and the token as it arrives, so a run that dies or loses the lock reply leaves what recovery needs (Issue 14).
PVE 8. PVE 8 has no SDN lock, so PVE does not serialize concurrent controllers on one cluster. When PVE answers the lock request as a missing endpoint (HTTP 501 or 404), CertiStack works without a lock for that run: it reads the cluster's pending SDN changes itself, applies nothing over one that is not its own, and relies on the host lock per node and the VNet ownership marker.
Pre-provisioned sandboxes. With lifecycle: preprovisioned, the platform operator provisions one dedicated sandbox ahead of time: a VNet with no gateway, no SNAT and no physical uplink. The run adds its own node-side firewall table, which drops forwarding, and a guard checks it every 2 seconds (see Issue 38). CertiStack validates its zone, VNet, subnet, no-gateway and no-SNAT invariants with read-only SDN requests and keeps it at teardown. It still adds and removes the host-side probe address and destroys every test VM, overlay and PBS map it owns. The report records a validated-and-retained sandbox, not a purged one. Keep the sandbox exclusive: the host lock serializes runs on one node, not on several, and CertiStack does not check what else is attached to the VNet. It credits a wire probe to whatever answers on the target address, so another guest on the VNet could answer in place of the VM under test.
Forwarding. PVE's ifupdown2 turns IPv4 forwarding on for every SDN bridge at each network reload, and CertiStack refuses a sandbox VNet that forwards. For an ephemeral sandbox the run turns forwarding off itself once the node's reload has brought the bridge up, and the containment table installed with the probe address drops forwarded packets even if a later reload turns it back on; the run also checks the switch every 2 seconds and turns it off again (see "The probe address"). A pre-provisioned VNet is the platform's, so the platform keeps forwarding off, for example with an /etc/network/if-up.d/ hook that writes 0 to /proc/sys/net/ipv4/conf/<vnet>/forwarding on every node that may run a validation. Otherwise the run refuses it before it adds the probe address. During the run it puts back only a forwarding switch that was off when it started and a reload turned on; any other setting of the platform's bridge (IPv6 forwarding on a bridge the platform leaves on, for example) is left as it was found.
The probe address. probe_ip is a temporary address the worker assigns to the VNet bridge for Stage 2 probes. It adds only the connected subnet route, is removed during teardown and recovery, and is never a gateway or SNAT address. Wire probes (tcp, http, dns, ldap, mssql, smb) are sent from it and bound to the VNet interface, so neither the host routing table nor a host listener can answer them. A network reload on the node (PVE makes one at every cluster-wide SDN apply, whoever applies) removes the address, which is not in the node's network configuration; the run checks the address, its containment table and the bridge's forwarding every 2 seconds and before each wire probe starts, and puts back what is gone, with a warning that names the cause. ip_range must not overlap loopback, unspecified, link-local, multicast or reserved space, which the worker host would answer itself.
Choosing the range. The rule: ip_range must be a network in which the PVE node that runs the test has no address and no route. The addresses in it are the sandbox's, the ones the guests will have: a guest that recovery adapts (the default mode) is given an address from the range, so the range need not be the guest's production subnet, and any range your node does not use will do. A guest that keeps its production address (network_recovery.mode: preserve, for example a Windows guest) needs ip_range to be its production subnet, so the node must not sit in that subnet: run on a node of the cluster that has no leg in it, or re-address the guest; where the node has a leg in it, the plan is refused. Use a range your node does not use. The examples use 192.0.2.0/24, a range reserved for documentation, and ip -4 route on the node lists the networks it does use. Write the range as a network address in IPv4 CIDR form: 192.0.2.0/24, not 192.0.2.5/24 (the plan names the network you meant), and not an IPv6 range (those are not supported). The sandbox is a disconnected network, but while a run lasts the probe address gives the node a connected route for the whole range, so the node's own traffic for any real network inside it would go into the sandbox. So CertiStack refuses, before it creates anything, a range that holds the PVE API (PVE_URL) or the PBS server (PBS_REPOSITORY), whose connection the run would lose. It also refuses a range that equals, contains or lies inside a network in any of the node's routing tables (the main table and the others, such as the table 52 where Tailscale keeps its subnet routes; every route but a default route) or the network of one of its interface addresses, that holds one of the node's own addresses, or that holds the gateway of any of its routes (the default gateway included). It also refuses a range that holds an address the node needs to keep reaching: the address your SSH session reaches the node from (the controller reads it in its own session and tells the worker), a DNS server of the node's /etc/resolv.conf, or a peer of a vxlan zone (network.peers), each with a message of its own. A probe_ip that is an address of the node is refused too. certistack plan lists the two checks first (sandbox-range and sandbox-network), shows a failed one as a ✗ line with the steps after it not reached, and exits non-zero; certistack check <plan.yaml> makes them its layer 8, certistack doctor shows them as sandbox_range and sandbox_network, and a run makes them again before it creates anything. When the node's routes cannot be read (or are more than what is read of a table) the step says could not check the node's routes: in plan, check and doctor, which change nothing, that is a warning, not a pass, so compare the range with ip -4 route yourself; a run refuses to start, naming the cause and how to retry, because it would otherwise go ahead on a range it could not check. For a pre-provisioned sandbox the plan's own VNet interface is left out, since its range and probe address live there; a run in progress on the node, or one that did not finish, shows up as an overlap on its own VNet interface. Each refusal is explained in Issues 39 to 44.

`tiers` and the dependency order

Tiers define the boot order of your application, in ascending level. CertiStack starts each VM in tier N, waits startup_grace_period_sec, and runs the Stage 2 and Stage 3 probes of every VM that reached Stage 1. A failure does not hide its independent peers: every current-tier outcome, and a failure framebuffer per affected VM, goes into the report. Tier N+1 begins only after all VMs and probes in tier N pass, and a failed tier blocks the tiers that depend on it. With soak_time_sec, the tier is then held and its services are checked again (see "Tier soak: what it checks").

Snapshot selection

Set each VM's source_vmid and source_pbs.snapshot: "latest" for normal scheduled validation. Before any SDN or VM resource is created, CertiStack queries that VM's PBS backup group, selects the highest backup-time, and replaces latest with the exact snapshot path, which the report records. A full snapshot path is for an operator who needs to replay a historical recovery test.

source_vmid names the protected VM in the report, so it must be the VM whose backup group the snapshot belongs to: source_vmid: 100 with snapshot: vm/200/... is refused. A VM that omits source_vmid is reported under the VM whose backup its resolved snapshot is. A VM that leaves out source_pbs.storage_id (or all of source_pbs) gets defaults only for what it leaves out: a snapshot it pins and a disk it names are never replaced.

Source disk layout

Use source_pbs.drives for every backed-up disk. Each entry keeps its source PVE slot on any bus (scsi0–scsi30, virtio0–virtio15, sata0–sata5, ide0–ide3) and names the PBS archive. Leave out Cloud-Init CDs and other media that are not in the backup. source_pbs.drive (one disk) is still accepted; its slot is the one its archive names (drive-virtio0 is virtio0), else scsi0.

Every run reads the snapshot's qemu-server.conf, whatever the plan inherits, and applies what a restore cannot do without. A real configuration is a few KiB: a snapshot whose listing gives the configuration blob more than 1 MiB, or one that restores to more than 1 MiB, is refused without being parsed. The table says what the sandbox VM gets and where the report records it.

Case	What the sandbox VM gets	Report
Boot-critical hardware	Firmware (see "UEFI and TPM firmware state"), machine type, SCSI controller and guest OS type come from the backup unless the plan sets them. A guest may not boot on PVE's defaults: a Windows guest installed on an LSI controller has no driver for virtio-scsi.
Machine version	The backup's setting, as `qmrestore` keeps it. PVE resolves an unversioned type at every start: for a Windows guest, on the QEMU version that created the VM (its `meta` line), or on 5.1 when that QEMU is older than 9.1. PVE writes the sandbox VM's `meta` itself and refuses to change it, so when the backup's version differs from the node's latest, the sandbox VM is pinned explicitly to the version a restore would start on.	`restored_hardware.running_machine`
A machine version the node cannot run	The VM is refused before anything is prepared. A backup made on a newer Proxmox VE can pin a versioned machine (`pc-q35-11.0+pve2`) that the node's QEMU is too old to run, and PVE fails the start ("Installed QEMU version … is too old to run machine type …") only after the guest is prepared. `plan` and `run` compare each VM's versioned machine (`pc-q35-N.M` or `pc-i440fx-N.M`, with or without `+pveK`) with the node's machine list on `N.M`, and refuse the VM, naming the machine and the newest versions the node offers. Unversioned `q35`, `pc`, `i440fx` and no setting are never refused. Restore on a node whose QEMU is as new as the one that made the backup, or set `machine:` for that VM to an unversioned type. A machine list that cannot be read is a warning.
SMBIOS identity	The backup's `smbios1` (UUID and any manufacturer, product, serial, SKU and family strings), as `qmrestore` keeps it; a guest can tell a new SMBIOS UUID from its own. The VM generation ID is new, as after any restore.	`restored_hardware.smbios_uuid`
CD-ROM drives	Kept but empty, so the guest finds the same controllers and no configuration drive is regenerated with the source's network settings.	`restored_hardware.empty_cdroms`
No balloon device	A source with `balloon: 0` gets none, unless the plan sets `hardware_overrides.balloon_mb`: a device the guest never had is new hardware to it.	`restored_hardware.no_balloon`
CPU weight	The plan's `hardware_overrides.cpu_weight`, else the backup's `cpuunits` under `use_backup_config`, else 50. A larger weight changes how fast a guest boots and answers when the node is busy.	`restored_hardware.cpu_weight` (absent only when no weight was sent and PVE applied its default)
CPU model, and settings that are not replayed	CPU type `host`, the node's own CPU, whatever the source's `cpu:` line named (a model such as `x86-64-v2-AES`, extra flags, `hidden=1`, `numa`, `vcpus`). A guest that checks its CPU model, or was set up on a feature set the node's CPU lacks, can behave differently in the sandbox, and a passing run does not show that it works on the source's CPU. The same holds for what a sandbox always sets for itself: the QEMU guest agent is enabled whatever the source said (so `qga` probes can run), and each disk is attached as a CertiStack overlay, so a source disk's `serial`, `wwn`, `ssd`, vendor and product strings, cache and aio modes are not carried. Anything your acceptance depends on beyond this table needs a wire or guest probe of its own.	`restored_hardware.cpu`
Boot disk and boot order	The first restored disk in the source's boot order (`boot: order=...`, or the legacy `bootdisk`); without one, the disk PVE would pick, first by bus (`ide`, `scsi`, `virtio`, `sata`) and index. The whole boot order is the source's order restricted to the restored disks, in the source's sequence: `order=scsi0;scsi1` stays so, and `order=scsi1;scsi0` keeps the second disk first. The firmware shows a guest only the disks in the boot order, so a guest that reads `/boot` from a second disk needs it. Entries that are not restored disks are left out: a CD-ROM, `net0` (a restored VM never boots from the network), a disk the plan omits, and a UEFI guest's `efidisk0` and `tpmstate0`. When the source order names none of the restored disks, the VM boots its boot disk alone.	`restored_hardware.boot_disk`, `restored_hardware.boot_order`
A disk the backup does not contain	The plan is refused.
Disks the plan leaves out	A plan may restore only some disks, for example the boot disk alone. The run warns, so a partial restore is never presented as the whole VM. Each archive must go to the slot it was backed up from, and only once.	`omitted_source_disks`
Disks excluded from the backup (`backup=0`)	No restore can include them, so the evidence says the VM was never whole in PBS.	`excluded_source_disks`

Backup protection

source_crypt_mode is the weakest PBS crypt mode among the VM's disk archives (encrypt, sign-only or none), and source_signature_verified is true when the configured key verified the snapshot manifest's signature. A key does not by itself require signed backups: it is used only where the snapshot's file list says the backup is signed or encrypted, so an unsigned backup still restores, as unsigned, and says so. A VM that must not depend on that sets require_signed_backup: true. It is then refused before anything is created (and doctor fails it) unless CERTISTACK_PBS_KEYFILE is set and its key verified the manifest's signature. That does not prove where the backup came from beyond holding the key, and the backup log (see "Backup consistency") is not signed.

Backup consistency

source_consistency says how consistent the backed-up disks were, from the backup's own log (client.log.blob, which vzdump stores with the snapshot):

quiesced: the guest agent froze the file systems (VSS on Windows);
crash-consistent: a running VM was backed up without a freeze that held (no agent, a refused freeze, for example by SELinux, or Windows released it before the snapshot);
powered-off: the VM was not running, or was shut down for the backup;
unknown: the log could not be read, or does not show how the snapshot was taken. Only a refused or failed freeze is conclusive on its own. Every other state needs the log to reach vzdump's started backup task line, and quiesced needs the freeze, the thaw and then that line, in order. vzdump writes issuing guest-agent 'fs-freeze' command before it tries the freeze, so that line alone proves nothing. Only the first 1 MiB of the log is read.

source_consistency_detail gives the reason in the log's words, and a crash-consistent backup is flagged in the run's progress. It fails a VM only when the plan sets require_quiesced_backup: true on it: then anything but quiesced or powered-off, unknown included, fails that VM before it is restored, and doctor fails it too.

The log is read through the PBS API with the configured repository, credentials and certificate fingerprint, because proxmox-backup-client cannot fetch a file outside the snapshot manifest. The manifest's signature does not cover it. The log of an encrypted backup is encrypted with the backup's key. CertiStack downloads it raw and decrypts it (AES-256-GCM, with the zstd compression PBS applies) with CERTISTACK_PBS_KEYFILE and, for a protected key, PBS_ENCRYPTION_PASSWORD, so an encrypted backup can meet require_quiesced_backup. What that establishes:

Only an encrypted blob is accepted. AES-GCM under the backup's key authenticates it, so something that holds the key wrote it. A plaintext blob for an encrypted backup needs no key to write, and is unknown, never quiesced.
AES-GCM binds the log to no snapshot, so the log must also say it is this snapshot's own: vzdump's first line names the VM (Starting Backup of VM N) and the archive line names the snapshot (creating Proxmox Backup Server archive 'vm/N/<time>'). A log that names another VM or backup, or neither, is unknown.
It does not prove the text is true (a holder of the key could write a false log), nor that the log is the newest word about the snapshot.
A log that does not decrypt, or is larger than 8 MiB raw, is unknown, with the reason.

Backup configuration inheritance

use_backup_config: true has CertiStack read the exact resolved snapshot's qemu-server.conf and discover every disk image archive (drive-scsiN, drive-virtioN, drive-sataN, drive-ideN). The inherited allowlist is CPU cores and sockets, memory and ballooning, CPU weight, machine type, guest OS type, firmware and SCSI controller. The CPU model is not on it (see "Source disk layout"). Explicit plan values override the inherited ones, field by field. The source net0 MAC and NIC model are kept too, attached only to the isolated VNet, so a guest network profile bound to a MAC or a driver keeps working.

CertiStack never replays source network interfaces, PVE tags, hooks, startup rules, host PCI or USB passthrough, or arbitrary QEMU arguments. Source tags are kept as attestation evidence only; the sandbox gets a fresh isolated NIC and only CertiStack ownership tags.

UEFI and TPM firmware state

A UEFI source VM (bios: ovmf) keeps its firmware state in two small disks: the UEFI variable store (efidisk0: boot entries and Secure Boot keys) and, with a virtual TPM, the TPM state (tpmstate0). CertiStack restores both with every plan, whether or not it sets use_backup_config, because a UEFI VM restored without them, or under SeaBIOS, is a different machine.

Each image is restored whole from PBS into a new raw volume on storage.overlay_storage, which UEFI VMs therefore require: the firmware writes its variable store, and PVE keeps TPM state only in raw volumes. The protected backup is never written.
The PBS client checks every chunk while restoring. The report lists each image under source_disks with its archive and SHA-256, and under restored_hardware.firmware_disks with its options, for example efidisk0 efitype=4m,pre-enrolled-keys=1 and tpmstate0 version=v2.0. Test mode restores them fully too. Only the options that describe the image are carried over: efitype, pre-enrolled-keys and ms-cert for efidisk0, and version for tpmstate0.
A plan that leaves bios empty restores a UEFI backup under OVMF; bios: seabios for a UEFI backup is refused. A firmware disk excluded from the backup (backup=0) fails closed, because the source firmware state cannot be reproduced.
The copies hold the source's firmware secrets: Secure Boot variables and TPM-sealed keys such as a BitLocker protector. They are 0600, journaled like every overlay, and removed with the sandbox VM, including by crash recovery.
Data disk overlays carry backup=0, but PVE has no such flag for efidisk0 and tpmstate0. A cluster backup job that covers all VMs and runs during a validation therefore copies the sandbox VM's firmware state, TPM secrets included, into a new backup group. Exclude the sandbox VMIDs from such jobs (--exclude, or pool-based selection) or schedule validations outside the backup window.

Integrity scan

Before each VM starts, a full-integrity run reads all of its disks through the page cache. The scan (and copy-before-boot's copy) passes a disk only when the mapped image is exactly the size the snapshot's manifest records for that disk's image (what proxmox-backup-client snapshot files lists for the drive-<slot>.img.fidx archive), the whole image is read with no clean end of file earlier, and its size is the same afterwards. A map that is short or long, or that changes size during the scan, fails the run with the numbers instead of being reported as verified.

The size= of the drive in the backup's configuration is not the standard: PVE records the size a resize asked for, storage rounds a volume up (LVM to 4 MiB extents, ZFS and RBD to 1 MiB), and a disk resized or imported outside PVE leaves it stale (qm disk rescan refreshes it). vzdump backs up the volume's real size, so a configuration that differs from the manifest is normal; the evidence detail records both numbers and the run goes on. Only when the manifest lists no size for a disk does the configuration's size matter: the map must then be at least that long, and the evidence says the manifest gave no size. When neither gives a size, only the read of the whole map is checked, and the evidence says so.

Copy-before-boot

By default (copy_before_boot: auto), CertiStack copies a tier's disks to local overlay storage during the integrity scan when they would not all stay cached until boot, because reading directly from PBS would make parts of the guest's boot miss service start deadlines. The sandbox then boots from local copies, as a full restore does. When a tier's disks together are more than the node can keep cached, auto copies every VM of the tier, the ones that fit as well: a VM that runs from the backup would otherwise lose its cached blocks to the later VMs' disk reads. auto copies only when overlay storage is on a disk separate from the cluster database and has room for the disks plus a 5 GiB reserve. always copies unconditionally or fails the VM before boot, and never boots straight from the read-only PBS mapped devices through COW overlays. storage.max_copy_gib (default 0, uncapped) caps the total copied in one run. Reports record copied_before_boot and each disk's boot_source.

Guest network recovery

For a VM with Stage 2 wire probes, CertiStack writes one temporary connection profile into the VM's disposable COW overlays before it boots, so a Linux guest that binds its network to an interface name or a PCI path still comes up on the sandbox address. You normally need no network_recovery stanza: omitted means auto. What it reads and writes, how to set interface_name, sandbox_ip and attach_all_nics, and what to do for Windows and other guests are on Guest network recovery.

NIC MAC addresses

Every sandbox NIC is on one isolated VNet, and earlier tiers stay up while later ones start, so a MAC address must be one an interface can hold and must belong to one NIC of the plan. One function (internal/macaddr) applies the rules where an address enters: the plan loaders (the controller's and the node worker's), the PBS profile and layout parsers, backup resolution, and the PVE create request.

Usable. Six octets that name one interface: not all zeros, not the all-ones broadcast address, and not a multicast address (the lowest bit of the first octet is 0, so the first octet is even). Linux refuses to give an interface any of those (Cannot assign requested address), so a guest or hypervisor given one would fail after the restore. A plan whose mac is one is refused when it is validated. A backup whose net0 holds one is refused when its configuration is read (with use_backup_config); without use_backup_config it is read as having no net0 MAC, and an automatic wire-probe run then stops with "has no net0 MAC". A source net1 that holds one is reported as omitted, or refused when attach_all_nics asks for it.
Unique. No two NICs of the plan hold one address, in any spelling (case, :, - and dotted forms are the same address): not net0 of two VMs, in one tier or two, and not a VM's net0 and an attached net1. With one address on two ports, the VNet's bridge sends frames for both guests to whichever port spoke last, and a healthy guest becomes unreachable at random. The loader checks the plan's mac fields. The run checks again after the backups are read, because a MAC may come from a backup: backups of VMs cloned from one another carry the same MAC, and a source's net1 can repeat net0's. The error names both NICs (VM 9901 net0 and VM 9900 net0 both use 00:11:22:33:44:55). Give one of the VMs a distinct mac (in a mode other than preserve), or restore them in separate runs. certistack plan shows each sandbox VM's net0, net1, ... as the run will send them, and refuses a plan whose NICs share an address.
Preserve keeps the backup's identity. In network_recovery.mode: preserve nothing rewrites the guest's network configuration, so a profile bound to the adapter (an ifcfg HWADDR, a NetworkManager, Netplan or systemd-networkd MAC match, a Windows static adapter) finds its NIC only if net0 keeps the backup's MAC and model. The VM therefore takes the backup's net0 MAC and model when the plan sets none, whether or not use_backup_config is set, and a mac or nic_model that differs from the backup's is refused, naming both. Leave them out, or set them to the backup's own. Every other mode rewrites the overlay's profile to follow the plan's identity, so a different mac is allowed there. A source with no usable net0 has no identity to keep, and the plan's stands. Two preserved clones of one VM would also share their guests' addresses, so restore them in separate runs.

PVE writes an address as six upper-case hex pairs separated by colons. The request and the report's restored_hardware.mac use that spelling whichever one the plan used.

Recovery time in the report

For a passing validation, rto_seconds and phase_timings.total_rto_sec measure the time from the run start through the last verified recovery probe. Teardown is excluded and reported separately as teardown_sec. A failed validation has no achieved recovery point, so those fields measure the time through terminal failure, including teardown when cleanup runs before the result is finalized: compare failed-run cleanup with teardown_sec, not with a recovery objective.

"Boot duration" begins at the sandbox VM start request and ends at Stage 1 hypervisor readiness. Storage mapping, guestfs work and the full mapped-image integrity verification are timed separately. Each phase's duration in phase_timings is recorded whichever way the phase ends, so a run that failed or was cancelled in a phase reports the time it spent there, not zero. A failed run's total_rto_sec is its named phases plus the small bookkeeping between them (journal writes, tracking of created resources). A passing run's total ends at its last verified probe, so the tier soak and teardown are not in it.

What a report records of its policy

A pass is a pass under a recovery time objective and a set of host admission limits, so a report (schema 1.7 and later) records the ones it was judged by beside the result (and a signature, when there is one, covers them). verify-report prints them, --expect-plan holds them to the plan, and the controller refuses to write a report that records others. Reports and verification lists the members. What the plan decides:

rto_target_seconds is the plan's compliance.rto_target_seconds. It is absent when the plan sets none: a report of schema 1.7 or later without it had no objective, and a pass records an rto_seconds within it. A plan cannot give 0: it is refused, because zero would read as "not set".
The plan digests (validated_plan_sha256, resolved_plan_sha256, plan_digest_format: worker-plan-yaml-v1) are the SHA-256 of the canonical worker YAML before and after backup resolution. All worker policy and resolved hardware are included; aliases and the compliance.sign_key_path are removed. --expect-plan checks the validated digest when present, and backup-derived resolved values are the node's observation. Earlier reports can omit them.
admission.recovery_reservation_mib is the memory the plan's sandbox VMs reserve at the peak (every tier stays up until teardown, so the last tier boots beside all the others). Each VM counts its configured memory (2048 MiB when the plan gives none) and 256 MiB of host overhead. With use_backup_config: true a VM without memory_mb takes its memory from the backup on the node, which the controller and verify-report cannot recompute from the plan: they hold the figure to a floor (the memory the plan does give and the overhead of each VM), and the other members exactly.
admission.evaluation is the last admission check, absent when the run ended before admission was evaluated. Its decision is granted, denied_ram, denied_iowait, denied_iowait_unmeasured, denied_projected_ram, never_admissible, preflight_denied_memory or metrics_unavailable. host_ram_percent and host_iowait_percent are read from /proc/meminfo and from one half-second sample of /proc/stat. projected_ram_percent is (host_total_mib - host_available_mib + recovery_reservation_mib) / host_total_mib, the figure held to max_host_ram_percent. A measurement the check did not reach is 0, and the decision says which were taken. They are the node's own observations: nothing can recheck them afterwards, so a verifier holds them only to the report's own limits and arithmetic. A memory preflight refusal records required_available_mib with decision preflight_denied_memory, and never_admissible means the reservation exceeds the total-memory limit. These checks do not sample I/O wait or claim a full admission grant.

3. Probes

Each probe runs within its timeout_sec, retrying as retries allows. That budget is a hard boundary: a pass that arrives after it, or after the run is interrupted, does not count, and the probe fails. Size timeout_sec for the slowest start you accept; a Windows guest agent can take minutes to come up. The same holds for Stage 1: a QMP running status that arrives after boot_timeout_sec fails the VM.

What a probe result records

A pass means what the assertion and the policy it ran under say, so each result in a VM's probe_results records both: the plan's timeout_sec, retries, require_qga and server_name, and where and from what source the probe ran. Report schema 1.6 added these members; Reports and verification lists them all. Three need explaining:

attempts counts executions. The tcp and http probes, and dns over UDP, also poll inside one execution, so for them duration shows how long the service took to answer. For a probe that checks once per execution (qga, qmp, ldap, mssql, smb), more than one attempt is a service that failed checks before it answered.
server_name absent means the plan named none, so the probe asserted the address itself: no SNI is sent for an IP address, Host is the address and port, and the certificate had to carry the IP.
source_interface and source_address are the VNet interface and network.probe_ip the probe's socket was bound to: the source a guest firewall sees. A probe that cannot use that source fails rather than leave from another.

restored_hardware records the hardware the sandbox VM was created with. The report itself records plan_sha256, the SHA-256 of the plan file exactly as the controller read it (see Verify a report).

None of these members holds a credential, a guest's data or a screenshot. A screendump probe keeps only the digest of its image; the image is removed from the node's scratch directory when the probe ends (a copy that cannot be removed then is removed at teardown, and a failure to remove it is reported). The digest tells captures apart but lets no one view or verify a picture afterwards, and a screendump never shows that a guest works.

Type: `tcp` (Stage 2: wire-level TCP handshake)

Completes an active TCP three-way handshake from the node worker's temporary VNet address to the guest IP and port. Stage 2 TCP, HTTP, DNS and LDAP evidence proves worker-to-guest reachability, not an in-guest client path; each result records its execution origin. Use a qga probe when the assertion must run inside the guest.

- type: "tcp"
  target_ip: "192.0.2.10"
  port: 5432
  timeout_sec: 15
  description: "Verify PostgreSQL socket listener on port 5432"

Type: `http` (Stage 2: wire-level HTTP or HTTPS health endpoint)

Sends an HTTP GET, verifies the TLS handshake and asserts the response status. expected_status is an HTTP status code from 200 to 599: HTTP has no status above 599, and a 1xx status is never the final answer to the one GET the probe sends, so a plan that asserts one is rejected. A status line whose code is outside 100 to 599 is not an answer, whatever the plan expects.

- type: "http"
  target_ip: "192.0.2.30"
  port: 443
  path: "/healthz"
  tls: true
  server_name: "service.example.com" # optional: TLS SNI and HTTP Host header
  expected_status: 200
  timeout_sec: 20
  description: "Verify Web App HTTPS healthz endpoint returns 200 OK"

Each connection attempt (connect, TLS handshake and waiting for the response headers) gets at most 3 seconds, whatever timeout_sec is. timeout_sec bounds how long the probe keeps trying, and an endpoint that always needs more than 3 seconds to answer does not pass. Give such an endpoint a cheaper health path.

path is the request target, sent as written. It may carry a query (/health?check=database), and percent-encoding such as %2F reaches the server unchanged. A path with no percent-escapes may hold a space or a non-ASCII character, which are sent percent-encoded (%20, the UTF-8 octets). What cannot reach the server as written is rejected: a #fragment, a space in the query, a % that starts no escape, and a path that has percent-escapes and also a character that needs encoding (write it fully percent-encoded).

The url shorthand (url: "http://192.0.2.30:8080/health?check=database") stands for target_ip, port, path and tls, and its path and query are kept as written too. A port, path or tls: true set beside url must agree with it. User information (user:password@) is rejected, and so is any scheme but http and https.

HTTPS probes always verify the certificate chain and hostname: the recovered service certificate must chain to a CA that the node worker trusts, and its SAN must include server_name (or the target IP when no server name is supplied). The handshake needs TLS 1.2 or later, and no setting lowers that; a guest that offers only TLS 1.0 or 1.1 (an old Windows Server's IIS, for example) fails the probe. Show that such a service answers with a tcp probe, and say in the description that the TLS version is not asserted. insecure_skip_verify and tls_skip_verify are rejected on purpose: a recovery assertion must not rest on an unauthenticated TLS endpoint (the mssql probe's TDS 8.0 strict path, described under that type, asserts only that the SQL Server engine answers and says in its result that the certificate was not verified). On an isolated network, install your own CA on the node worker instead.

Type: `dns` (Stage 2: wire-level DNS resolution)

Sends one DNS query directly to target_ip, over UDP (re-sent every 2 seconds until the timeout) or TCP. The worker's own /etc/hosts and resolver configuration are never consulted.

Over UDP, a guest that is up but whose DNS server is not listening yet refuses the datagram (ICMP port unreachable), and one that is not up yet leaves its address unresolved (no route to host). Neither ends the probe: they count as "no answer yet", and the query is re-sent until the timeout, when the error names the last refusal. Over TCP a refused connection fails that attempt at once, and retries decides whether another follows.

With domain, the probe passes only when the server answers NOERROR with at least one A or AAAA record for that name. NXDOMAIN, REFUSED, SERVFAIL or an empty answer fails, so a restored DNS server that lost its zone is detected. The probe asks for A records and, when the name has none, for AAAA records.
Without domain, it is a listener check: the server must return any well-formed reply to a root NS query.

Every reply must answer the question asked: the same transaction ID, the question repeated with the same name (compared case-insensitively), type and class, and every record well formed, with name compression that only points back to earlier names. Only records of the asked type in class IN count, and only for the queried name or, when it is an alias, for the name its CNAME chain ends at (the report shows that chain). A reply must also be complete: the question and every record its header counts must be present, with no bytes after the last. A malformed, cut-short or padded reply fails, in listener mode too. Only an error reply (any RCODE but NOERROR) may leave the question out. A reply that is cut short and says so with the TC bit is the exception: its records up to the cut count, and transport: tcp avoids the cut.

- type: "dns"
  target_ip: "192.0.2.10"
  port: 53
  domain: "healthy.example.com"
  transport: "tcp" # optional; defaults to udp
  timeout_sec: 10
  description: "Verify internal DNS listener responds to queries"

With record_type: srv, the probe asks for SRV records instead and passes only when the answer holds at least one; the report lists each target, port, priority and weight. For a domain controller, set domain: "_ldap._tcp.dc._msdcs.example.com" (the DC locator record): it proves the restored DNS server loaded the AD-integrated zone, which it reads from the directory, and still carries the records clients use to find a DC. The records were written before the backup, so they do not show that Netlogon re-registered after the restore.

Type: `ldap` (Stage 2: wire-level directory probe)

Performs an LDAP v3 anonymous bind over TCP port 389. When base_dn is set, it also performs an anonymous base-object search for that DN.

The probe speaks cleartext LDAP only: it has no LDAPS and no StartTLS. A directory that enforces confidentiality answers the anonymous bind with confidentialityRequired (result code 13), and that fails the probe in every mode, listener check included. Only strongAuthRequired and inappropriateAuthentication count as a policy refusal that shows a server answered. Such a directory is not testable by ldap; a tcp probe on its LDAPS port (636) shows it listens, and says nothing about its directory.

Use an explicit DN when it is a business assertion that must exist; CertiStack does not weaken it if the restored directory advertises a different naming context. For a portable check across different directories, set base_dn: auto: CertiStack reads the server's Root DSE and requires a successful base-object search of one of its advertised naming contexts. Every advertised context is tried, in the order the server lists them, until one returns its entry or the timeout runs out; the same context in another spelling is tried once, and a failed exchange ends the attempts and says how many were not tried.

Omit base_dn only when an LDAP v3 listener and bind handshake alone is the intended assertion. It then passes when the anonymous bind succeeds, and also when the directory refuses it by policy with strongAuthRequired (a domain controller that requires signing) or inappropriateAuthentication (one that disallows anonymous binds): either shows that an LDAP v3 server answered, and the report says which. Any other bind result fails.

base_dn (other than auto) and naming_context are distinguished names in the syntax of RFC 4514, at most 1024 bytes, and a plan whose DN is not one is rejected before the run: an attribute type, = and a value in every RDN, a backslash only before a special character or two hexadecimal digits, and the characters " ; < > escaped inside a value. Spaces around , + and = are ignored.

A base-object search passes only when the directory returns the entry for that DN. DNs are compared as DNs, not as strings: case is ignored, escapes are decoded (\, and \2C are the same), and the assertions of a multi-valued RDN may come in any order (CN=J. Smith+OU=Sales is OU=Sales+CN=J. Smith). An attribute type written as its OID is not equated with its name, and internal spaces are not collapsed, so copy the spelling the directory returns. A search that succeeds with no entry (what a directory returns when it hides the object from anonymous clients) fails, and so does one that returns an entry for another DN. A bind or search result that is not a complete LDAPResult, or a Root DSE search that does not return one entry with the empty DN, fails.

- type: "ldap"
  target_ip: "192.0.2.10"
  port: 389
  base_dn: "auto" # portable Root-DSE discovery plus verified base-object search
  timeout_sec: 15
  description: "Verify any restored directory naming context is readable"

Active Directory. A domain controller refuses anonymous searches by default, so base_dn (explicit or auto) fails against it, and omitting base_dn proves only that something answers a bind. Set naming_context instead: CertiStack binds anonymously, reads the Root DSE (the one entry AD lets anonymous clients read), and passes only if the directory advertises that naming context, as defaultNamingContext or in namingContexts. No directory search is sent and the DC's policy stays as it is. The directory service answers the Root DSE itself, so a DC whose database did not open fails even though port 389 accepts connections. The report records the DC's dnsHostName and isSynchronized, which is informational: a DC restored without its replication partners may report FALSE while serving its directory correctly. naming_context works for OpenLDAP and 389-ds too (their Root DSE lists namingContexts), and cannot be combined with base_dn.

- type: "ldap"
  target_ip: "192.0.2.10"
  port: 389
  naming_context: "DC=example,DC=com" # Root DSE assertion; no directory search
  timeout_sec: 60
  description: "Verify the restored domain controller serves example.com"

Type: `mssql` (Stage 2: wire-level SQL Server protocol probe)

Sends a TDS PRELOGIN, the first message of every SQL Server connection, and passes only on a well-formed PRELOGIN response. No login is attempted and no credentials are needed, so the probe proves that the SQL Server engine answers its protocol, not merely that the port is open. The report records the server's version and encryption setting. The response must be structurally complete (a VERSION option of at least 6 bytes, every option's data inside the data area, no option repeated or overlapping another). A named instance must listen on a static port, because the probe does not query SQL Browser.

TDS 8.0 strict encryption is tested automatically. An instance set to Force Strict Encryption (SQL Server 2022 and later) expects a TLS handshake before any TDS message. The probe first sends a cleartext PRELOGIN, as TDS 7.x does. When the server answers that with a TLS record or drops the connection, the probe retries once, on a fresh connection, as TDS 8.0 strict: the TLS handshake first (ALPN tds/8.0, TLS 1.2 or later), then the same PRELOGIN inside the tunnel, read with the same checks. No plan field selects the mode. Both attempts share the probe's timeout_sec; a server that stays silent is not retried, and a TLS server that does not select tds/8.0, closes after the handshake or answers anything but a well-formed PRELOGIN response fails the probe.

On the strict path the server's certificate is not verified (a default SQL Server certificate is self-signed, and the probe has no way to learn which CA to trust), so a pass shows that something speaking the SQL Server protocol answered on that address and port, not whose engine it is. The cleartext path never authenticated the server either. The result's detail says "over TDS 8.0 strict encryption", the negotiated TLS version and "certificate not verified". This is the one place a probe does not verify a certificate: it is not a plan setting, and insecure_skip_verify is still rejected. As with every mssql pass, no login is attempted: a PRELOGIN response proves that the engine answers, not that a login or a query would succeed. For an assertion about the service itself you can still pair the probe with a qga probe (sc query MSSQLSERVER, or sc query MSSQL$<instance>, on Windows; systemctl is-active mssql-server on Linux). Do not weaken the instance's encryption policy to satisfy this probe.

- type: "mssql"
  target_ip: "192.0.2.20"
  port: 1433
  timeout_sec: 120
  description: "Verify SQL Server answers TDS on the restored guest"

Type: `smb` (Stage 2: wire-level SMB protocol probe)

Sends an SMB2 NEGOTIATE over direct TCP and passes only on a successful, complete and consistent NEGOTIATE response for one of the offered dialects (SMB 2.0.2 through 3.1.1). SMB 3.1.1 is offered with the negotiate context list MS-SMB2 requires (a preauthentication integrity context offering SHA-512), so a server that requires 3.1.1 (for example Samba with server min protocol = SMB3_11) is tested like any other, and a response that selects 3.1.1 must carry a well-formed context list naming SHA-512. The report records the dialect and whether the server requires signing. No session is set up and no share is opened, so the probe proves that the SMB server answers, not that a share or its data survived. For that, pair it with an application check that reads the data.

- type: "smb"
  target_ip: "192.0.2.21"
  port: 445
  timeout_sec: 60
  description: "Verify the file server answers SMB on the restored guest"

Type: `qga` (Stage 3: in-guest inspection through the QEMU guest agent)

Uses a constrained health-check vocabulary inside the guest operating system. Plan-authored commands are limited to hostname, guest-get-host-name, ping, guest-ping, systemctl is-active <service> and sc query <service>. Arbitrary shell commands and environment interpolation are rejected before execution. Guest output lands in the report, so the vocabulary stays read-only and narrow.

- type: "qga"
  # Debian and Ubuntu: the unit of the database cluster itself (16 is the
  # PostgreSQL version, main the cluster name; `pg_lsclusters` lists both).
  # On the RHEL family the unit that runs the server is `postgresql`.
  command: "systemctl is-active postgresql@16-main"
  expected_output: "active"
  require_qga: true # make a missing/unresponsive guest agent a failure
  timeout_sec: 30
  description: "Verify the PostgreSQL cluster unit is active inside guest"

Which unit you name decides what the probe proves. Some packages name a unit that is active whether or not the service runs: on Debian and Ubuntu, postgresql is a grouping unit that reports active with no database cluster running, so systemctl is-active postgresql passes on a guest whose database is down or was never restored. That is why the example names the cluster's unit, postgresql@16-main, which is active only while the cluster runs. CertiStack cannot tell a grouping unit from the unit that runs a service, so name the unit that runs it, and pair it with a wire probe to the service's port where one exists: that is what shows it answers.

A command passes only when the guest agent reports a normal exit with code 0 (and, when set, expected_output appears in its output). A command the agent reports as killed by a signal, or on Windows ended by an unhandled exception, fails with that signal or exception code in the report, whatever it printed.

hostname and guest-get-host-name ask the agent for the guest's host name instead of running a command, and expected_output must be a substring of it, matched case-sensitively: a Windows guest reports its name in upper case (WEB01), so expected_output: "web01" fails there. The answer has to be a host name: one that is empty, longer than 255 bytes, or made with control characters, characters that reorder or hide text, or invalid UTF-8 fails the probe. Guest text that reaches the report is cleaned the same way and cut to a fixed size (see Reports and verification).

On Windows guests, sc query <service> checks a service by its key name (for example W3SVC, LanmanServer or MSSQL$SQLEXPRESS). CertiStack runs sc.exe directly with the literal arguments query and the name, never through a shell, and passes only when the service's STATE is 4 RUNNING. A stopped, pending or missing service fails, and the report quotes the state or the sc.exe error. expected_output is optional and adds a substring check; a passing result names it (the output contained the expected "ACCEPTS_SHUTDOWN"), so two plans that require different substrings of one answer do not sign the same result. The name may contain letters, digits and $ _ . -. The Windows guest agent must allow guest-exec, which it does by default.

Fallback invariant: if QGA is not installed or unresponsive in the guest, a QGA probe without require_qga: true is marked SKIPPED and the VM's other probes decide pass or fail. A skip is never evidence: a VM needs at least one probe that is sure to assert something in the guest (see "Which probes verify a VM"), and plan validation refuses a VM whose every probe is an optional QGA probe. A probe stage in which every probe was skipped fails, and the VM is unverified.

A ping (or guest-ping) checks that the agent answers. Its reply is an empty object, so there is no output for expected_output to match: plan validation refuses an expected_output on a ping, and a probe built with one fails instead of passing without checking it. Use hostname, systemctl is-active <service> or sc query <service> when the answer must contain something.

Types: `qmp` and `screendump` (supporting evidence only)

qmp connects to the sandbox VM's QMP socket and passes only when query-status reports the VM running. Stage 1 already required that before any probe starts, so qmp proves only that the VM is powered on: a guest in a boot loop, at a "No bootable device" screen or at a kernel panic that does not stop QEMU is running. It is supporting evidence only and cannot stand alone: plan validation refuses a VM whose probes are all qmp or screendump (see "Which probes verify a VM"). Every VM needs at least one wire probe, or a qga probe with require_qga: true; a VM that offers neither a listening service nor a guest agent gives CertiStack nothing to assert, and a plan cannot include it. The alias qmp_status is accepted, and the result records its origin as hypervisor_qmp.

screendump captures the console framebuffer through QMP and passes when any image could be captured, whatever it shows, black included. It asserts nothing about what the screen shows; it is for the record, like the forensic capture CertiStack takes when any probe fails. A passing screendump records the SHA-256 of the image (image_sha256) and keeps nothing else: the image is not in the report and is removed from the node's scratch directory when the probe ends.

Neither says anything about what the guest did, so neither ever makes a VM verified. They stay useful beside a guest-level probe: the report records that QEMU ran the VM, and a failed run embeds a screenshot.

Which probes verify a VM

A VM is verified only if at least one guest-level assertion ran and passed:

a wire probe (tcp, http, dns, ldap, mssql, smb), which reaches a service in the guest over the sandbox network, or
a qga probe that was not skipped, which runs inside the guest.

A skipped optional qga probe, a qmp probe and a screendump probe never count. Plan validation refuses a VM with no probe that is sure to be one: only qmp and screendump probes, only optional qga probes, or those together. Add a wire probe, or set require_qga: true on a qga probe. A VM that reaches the probe stage without a guest-level assertion (for example because its optional guest-agent probes were skipped) is recorded as verification_status: unverified, fails the tier with the reason, and the evidence verdict (verify-report) rejects it.

Set require_qga: true for a test whose assertion is the guest-agent channel itself. An unavailable QGA channel is then a probe failure instead of a skipped optional inspection. The result records require_qga: true, so the report shows whether the plan required the agent even when it answered.

Tier soak: what it checks

tiers[].soak_time_sec holds a tier's passing VMs for that long before the next tier starts, so a recovered service has to stay up and not merely answer once. From report schema 1.7 the soak does that with the probes, not only with PVE's word that QEMU still runs the VM:

What is repeated. Every 30 seconds, counted from the end of the previous round of checks, and once more when the period is over, CertiStack runs each VM's service probes again: the wire probes (tcp, http, dns, ldap, mssql, smb) and the qga probes, the ones that assert something in the guest. qmp and screendump are not repeated; PVE is asked about the VM's state throughout instead.
One check is one run of those probes in the plan's order, each under the policy the plan gives it (timeout_sec, retries, require_qga), as in the probe stage. A tcp or http probe, and a dns probe over UDP, polls inside its timeout_sec, so an outage that ends inside that budget is not a failed check; one that outlasts it is. Size timeout_sec for the longest outage the service may have before you call it down. Samples between checks are not taken: a service that is down only between two checks is not seen.
Timing. A check that takes longer than the interval is followed at once by the next; checks never overlap. The last check starts when the period is over and may end after it, so a soak can last up to one check longer than soak_time_sec. The soak is not part of the RTO. A soak shorter than 30 seconds has the one check at its end.
A failed check fails the VM and the tier, at the first one: the VM is recorded as failed, is not checked again, and its peers in the tier are checked to the end of the period so their evidence is complete. An optional qga probe that was skipped in the probe stage stays skipped; one that answered then and is silent later is a failed check.
The VM's state. Every 10 seconds PVE is asked whether each VM still runs (in the startup grace period and while the probes run too). A read that fails is not evidence that the VM runs: a refusal (HTTP 401 or 403, the token lacks VM.Audit on the VM) fails the VM at once, since it cannot get better; any other failure (a timeout, a server error, a malformed answer, an answer that names no state) is waited through for two minutes after the last read that worked, and then fails the VM with "its liveness is unknown". A wait in which not one read worked fails its VMs too, however short it was. A VM that PVE says no longer exists is a stop.
What the report records. A VM's record carries a soak summary of fixed size: the soak's configured_sec and interval_sec, when it started_at and when its first and last checks ran, samples and failed_samples, for each repeated probe how many checks passed, were skipped or failed, its worst_duration and its first error (cut to 512 bytes), and status_reads, status_read_failures and first_status_error for PVE's state reads. verify-report rejects a passing VM whose summary is missing, failed, ended before the configured period, or saw no state read succeed (see Verify a report).

4. Literal plans and the protected environment

Plans are literal documents. CertiStack does not expand ${VAR_NAME} or shell syntax in a plan: a plan can be submitted through an API, kept as evidence and inspected by another operator, so environment interpolation would risk credential disclosure and make its effective content ambiguous.

Keep connection credentials in a protected, allowlisted environment file passed with --env-file. Its values are parsed literally, never shell-sourced. The file is UTF-8 (a byte order mark is accepted, UTF-16 is refused), and the error for a line that is not NAME=VALUE with a supported name gives the line number and quotes the name only when it looks like one, so a secret pasted without its name is not printed back. Keep plan identity explicit in the plan (an excerpt; examples/minimal-single-vm.yaml is a complete plan):

plan_id: "acme-erp-dr"
compliance:
  frameworks: ["DORA"]
  retention_days: 2555
tiers:
  - level: 1
    vms:
      - vmid: 9001
        source_vmid: 100
        source_pbs:
          storage_id: "pbs"
          snapshot: "latest"

If a site needs a family of plans, render them in a controlled build step, validate each generated file, and keep its SHA-256 (the report records the digest of the plan that ran as plan_sha256). Do not run a shell template against a secret-bearing environment.

5. Complete field reference

The parser rejects unknown keys, a plan larger than 10 MiB, and a file that contains more than one YAML document, so every key it accepts is listed here. An unknown key's error names its line and section, the key you probably meant and the keys that section accepts, for example line 4: unknown key "grace" in tiers[]; did you mean "startup_grace_period_sec"?. An alias is accepted for older plans and normalised to the canonical key; write the canonical key in new plans. A plan that gives a setting under both spellings must give the same value, or it is refused (a key written as 0 counts as given). A rejected key parses but fails validation, so an unimplemented or unsafe setting is never silently ignored. Credentials never belong in a plan: pbs_password, tokens and similar keys are unknown to the parser and fail to load.

Top level

Key	Type	Status	Rule
`version`	string	required	`v1`; `certistack/v1` is an accepted alias
`plan_id`	string	required	matches `^[a-zA-Z0-9_-]{1,128}$` (letters, digits, `_` and `-`, at most 128 characters: a longer one could not be kept by the evidence hub)
`name`	string	required	non-empty
`environment`	string	optional	free-text label recorded in the report and shown by `validate`
`use_backup_config`	bool	optional	inherit safe compute settings and all backed-up disks from each resolved snapshot
`compliance`	map	required	see below
`admission_control`	map	required	see below; `admission` is an accepted alias
`network`	map	required	see below
`storage`	map	optional, discouraged	see below
`tiers`	list	required	at least one tier

`compliance` keys

Key	Type	Status	Rule
`frameworks`	list of string	required	at least one item, each one of `NONE` (no standard; it stands alone), `DORA`, `DORA-Article-12`, `SOC2`, `SOC2-Type2-CC7.4`, `HIPAA`, `HIPAA-164.308-A7`, `ISO27001`, `ISO27001-A8.13`, `ISO27001-A.12.3`, `ISO27001-A12.3`, `CIS_V8`; recorded in the report; a label is not a certification, and the `CHANGE` placeholder from `init` is refused
`retention_days`	int	required	`1` to `36500` (100 years); a missing key or `0` is refused
`sign_key_path`	string	optional; used by the Enterprise Edition	Ed25519 private key file that the Enterprise Edition's `certistackd` service signs with; never sent to the worker. The Community Edition accepts it and ignores it (no existence, mode or format check): it writes unsigned reports
`rto_target_seconds`	float	optional	finite and `> 0`; leave the key out for no objective (a `0`, a negative number or `.nan` is refused: zero would read as "not set" and every run would pass it); when set, a run whose measured RTO exceeds it is reported as failed, and the report records the objective (`rto_target_seconds`) beside the measured `rto_seconds`

`admission_control` keys

Key	Type	Status	Rule
`max_host_ram_percent`	float	required	finite, `> 0` and `<= 100`; the most host memory in use, with the recovery reservation, that still admits the run; examples use `85.0`
`max_host_iowait_percent`	float	required	finite, `> 0` and `<= 100`; the most CPU I/O wait that still admits the run; `max_host_io_wait_percent` is an accepted alias; examples use `12.0`
`defer_retry_interval_sec`	int	required	`1` to `604800` (7 days); the wait between admission checks; examples use `30`
`max_defer_retries`	int	required	`1` to `10000`; the retries before the run is marked throttled and fails; examples use `10`
`max_concurrent_boots`	int	optional	`0` (unlimited, the default) to `64`; how many sandbox VMs may boot at once on the node
`boot_settle_sec`	int	optional	`0` (default `180`) to `3600`; only with `max_concurrent_boots`; how long a started VM counts as booting

`network` keys

Key	Type	Status	Rule
`mode`	string	required	`simple` (single-node bridges) or `vxlan` (multi-node clusters)
`lifecycle`	string	optional	`ephemeral` (default: CertiStack creates and removes the zone and VNet) or `preprovisioned` (a platform-owned sandbox, validated read-only and kept)
`zone_id`, `vnet_id`	string	required	PVE SDN identifier: a letter followed by letters and digits, at most 8 characters (for example `csZone01`, `csVnet01`). For an `ephemeral` sandbox the two must differ (CertiStack creates both and tracks their ids in one set); a `preprovisioned` sandbox names a zone and a VNet that already exist, and they may share an id
`tag`	int	required for `vxlan`	VNI `1..16777215`, explicit and unused by every existing PVE VNet; rejected for `simple` (its VNet carries no tag)
`peers`	string	required for `vxlan`	comma-separated IP addresses without duplicates, sent as the PVE zone `peers` field, none of them inside `ip_range`; rejected for `simple`
`ip_range`	string	required	IPv4 CIDR of the isolated subnet, written as its network address in exactly one spelling (`192.0.2.0/24`, not `192.0.2.5/24`, `192.0.2.0/024` or a range with spaces; IPv6 is not supported), outside loopback, link-local, multicast, and reserved space; `subnet_cidr` is an accepted alias. No address, route or default gateway of the PVE node, and neither the PVE API nor the PBS server, may lie in it: `plan`, `check`, `doctor` and the run refuse otherwise, because the probe address would capture the node's own traffic for that network (see "Choosing the range"; `ip -4 route` on the node lists the networks it uses)
`probe_ip`	string	required	address inside `ip_range` that is neither the network nor the broadcast address, and not an address of the node
`gateway`	string	rejected	must be empty; a sandbox never has a gateway
`ipam`	string	rejected	not implemented; must be empty
`mtu`	int	rejected	not implemented; must be unset

`storage`

Prefer the protected environment file (PBS_REPOSITORY, PBS_FINGERPRINT) over this stanza: a repository string can contain a token identity, and a plan is kept as evidence.

Key	Type	Status	Rule
`pbs_repository`	string	optional, discouraged	when set, a VM may omit `source_pbs`; the parts it omits default to `storage_id: pbs`, `snapshot: vm/<source_vmid or vmid>/latest` and, without `use_backup_config`, `drive: drive-scsi0`
`pbs_fingerprint`	string	optional, discouraged	PBS certificate fingerprint
`pbs_namespace`	string	optional	default for every VM's `source_pbs.namespace`
`scratch_dir`	string	optional	node-local private work directory (guest network recovery, screendumps; also overlays when `overlay_storage` is unset); the worker rejects shared temporary directories such as `/tmp`
`overlay_storage`	string	required on stock Proxmox VE	PVE storage ID of a file-based storage (`dir`, `nfs`, `cifs`, `cephfs`, `btrfs`) with `images` content. Overlays are created as `<storage>/images/<vmid>/certistack-delta-*.qcow2` and attached by volume ID. The run fails before any mutation if the storage is inactive, lacks `images`, or has under 5 GiB free, and it refuses an `images/<vmid>` directory that holds other volumes. When PVE's status read of the storage does not answer (another storage, often a PBS, is slow), the run checks the storage on the node instead and warns that PVE did not confirm it is active: it reads the storage's configuration from `/etc/pve/storage.cfg` (enabled, file-based, `images` content, this node, the path `pvesm` resolves) and, for a storage PVE expects mounted (`is_mountpoint`, or NFS, CIFS, CephFS, GlusterFS), requires the mount in the kernel's mount table. A directory beneath an absent mount is refused, never measured, and so is a storage whose configuration cannot be read. `--overlay-storage` / `CERTISTACK_OVERLAY_STORAGE` override it
`max_copy_gib`	int	optional	`0` (no cap, the default) to `1048576`. The most copy-before-boot may write in one run, all VMs together. `auto` does not copy a VM whose copies would pass it, and warns; `always` fails that VM. Use it where the free space is not all yours, such as a scratch disk shared with other tenants

`tiers[]`

Key	Type	Status	Rule
`level`	int	required	`>= 1`, unique across the plan, executed in ascending order; `tier` is an accepted alias
`name`	string	optional	defaults to `Tier <level>`
`startup_grace_period_sec`	int	optional	`0` to `604800` (7 days)
`boot_timeout_sec`	int	optional	`0` to `604800`; default 60; PVE start plus QMP readiness
`soak_time_sec`	int	optional	`0` to `604800`; hold the tier for this long after its probes pass before starting the next tier; recorded in the report timings. During the soak CertiStack watches that QEMU keeps each VM running and runs each VM's service probes (the wire probes and `qga`) again every 30 seconds and once more when the period is over; a VM that stops, or whose service fails a check, fails the tier. See "Tier soak: what it checks"
`vms`	list	required	at least one VM

`tiers[].vms[]`

Key	Type	Status	Rule
`vmid`	int	required	`100` to `999999999` (a Proxmox VM ID, the range `init` allocates from), unique across the plan; the temporary sandbox VM; `target_vmid` is an accepted alias, and when both are given they must be equal
`source_vmid`	int	required when `snapshot: latest`	the protected source VM; must be the VM of the snapshot's backup group; when omitted, the report takes it from the resolved snapshot
`name`	string	required	non-empty
`mac`	string	optional	six-octet MAC for the isolated NIC; normally inherited with `use_backup_config`. Not all zeros, not the broadcast address, not a multicast address (even first octet), and unique among every NIC of the plan, in any spelling; with `network_recovery.mode: preserve` it must be the backup's own `net0` MAC, and is inherited when left out (see "NIC MAC addresses")
`nic_model`	string	optional	`virtio`, `e1000`, `e1000e`, `igb`, `ne2k_pci`, `ne2k_isa`, `pcnet`, `rtl8139`, `vmxnet3`; with `network_recovery.mode: preserve` it must be the backup's own `net0` model, and is inherited when left out
`machine`, `ostype`, `bios`, `scsihw`	string	optional	explicit PVE values that override inherited ones
`network_recovery`	map	optional	see below
`source_pbs`	map	required unless `storage.pbs_repository` is set	see below
`hardware_overrides`	map	optional	`cores` (`1`-`256`), `sockets` (`1`-`16`), `memory_mb` (`16`-`4194304`), `balloon_mb` (`1`-`4194304`, not above `memory_mb`), `cpu_weight` (`1`-`262144`, the range of the PVE `cpuunits` option; PVE clamps the effective weight to `1`-`10000` on cgroup v2): the ranges a backup's own configuration is held to. `0` or a missing key inherits the backup's value (with `use_backup_config`) or takes the default; a negative or larger value is refused
`copy_before_boot`	string	optional	`auto` (default), `always` or `never`. Whether the integrity scan also copies the VM's disks next to its overlays, so the sandbox boots from local copies as a restore does. `auto` copies when the tier's disks would not all stay cached until the VM boots, unless overlay storage is on the cluster database's disk or lacks room for the disks plus 5 GiB; it warns when it can't. `always` copies or fails the VM. Reports record `copied_before_boot` and each disk's `boot_source`
`require_quiesced_backup`	bool	optional	default `false`; fail the VM before anything is restored unless its backup is `quiesced` or `powered-off` (see "Backup consistency")
`require_signed_backup`	bool	optional	default `false`; fail the VM before anything is created unless its backup manifest is signed (`sign-only` or `encrypt`) and the key in `CERTISTACK_PBS_KEYFILE` verified it (see "Backup protection")
`probes`	list	required	at least one probe, and at least one that is sure to assert something in the guest: a wire probe, or a `qga` probe with `require_qga: true`. `qmp` and `screendump` probes and optional `qga` probes (skipped when the guest agent does not answer) do not count; see "Which probes verify a VM"

`tiers[].vms[].source_pbs`

Key	Type	Status	Rule
`storage_id`	string	required	`pbs` (this run's configured endpoint); other selectors are rejected at run time. It may be left out when `source_vmid` or `storage.pbs_repository` is set: then only the parts of `source_pbs` the VM leaves out default, and a given snapshot or disk is kept
`snapshot`	string	required	`latest`, `vm/<vmid>/latest`, or `vm/<vmid>/<RFC 3339 time>`; its `<vmid>` must be `source_vmid` when both are given
`drives`	list	required unless `use_backup_config`	entries of `slot` (`scsi0`..`scsi30`, `virtio0`..`virtio15`, `sata0`..`sata5`, `ide0`..`ide3`; unique) and `archive` (PBS archive filename without path separators); every slot must be backed up in the snapshot
`drive`	string	deprecated alias	a single archive, in the slot it names (`drive-virtio0`) or `scsi0`; beside `drives`, it must be one of them
`namespace`	string	optional	PBS namespace; defaults to `storage.pbs_namespace`

`tiers[].vms[].network_recovery`

How recovery works is on Guest network recovery.

Key	Type	Status	Rule
`mode`	string	optional	`auto` (default), `preserve`, `ifcfg`, `networkmanager`, `netplan`, `systemd-networkd`, `ifupdown`
`sandbox_ip`	string	optional	usable IPv4 address in `network.ip_range`, unique per plan; required when a VM's wire probes target more than one address; forbidden with `preserve`
`interface_name`	string	optional	Linux interface name for a legacy name-bound profile: 1 to 15 characters from letters, digits, `_`, `-` and `.`, and not `lo`, `.` or `..` (the kernel refuses the last two). The plan and the guest-network adapter apply the same rule
`capture_diagnostics`	bool	optional	record guest-side diagnostics on a wire-probe failure
`attach_all_nics`	bool	optional	also attach the source's `net1`+ NICs (MAC and model only) to the isolated VNet; their MACs must be usable and distinct from every other NIC of the plan, `net0` included (see "NIC MAC addresses")

`tiers[].vms[].probes[]`

Key	Type	Status	Rule
`type`	string	required	`tcp`, `http`, `dns`, `ldap`, `mssql`, `smb`, `qga`, `qmp`, `screendump`; aliases `tcp_handshake`, `http_get`, `qga_ping`, `qmp_status`
`timeout_sec`	int	required	`1` to `604800`; a missing key becomes 10, but a `0` or negative value is refused, not turned into 10. Recorded in the probe's result as `timeout_sec`
`retries`	int	optional	`0..10`; 0 keeps automatic polling until the timeout. Recorded in the result as `retries`, with the `attempts` the probe took
`description`	string	optional	defaults to `<type> verification probe`
`target_ip`	string	required for `tcp`, `http`, `dns`, `ldap`, `mssql`, `smb`	literal IP inside `network.ip_range`, never `probe_ip` or the subnet network or broadcast address, and not an address of the worker host (for example another host address on a preprovisioned VNet). Every wire-probe target belongs to exactly one VM of the plan, whatever its `network_recovery.mode`, so no two VMs name the same target. The loaders, `certistack plan` and the run (before it creates anything, and again before each VM's probes) apply the rule, and the controller and `verify-report` apply it to a report's records: a report in which passing wire probes of two VM records name one address is not signed, and is rejected as pass evidence even without `--expect-plan`. It checks what the plan and the report say, not which guest answers (a run does not read the VNet's neighbour table), which is why a preprovisioned VNet must not be shared (see `lifecycle`)
`url`	string	optional (`http`)	`http(s)://host[:port]/path[?query]`; expanded into `target_ip`, `port`, `path` (with the query, exactly as written), and `tls`; no user information or `#fragment`; a `target_ip`, `port`, `path`, or `tls: true` also set must agree with it, and a `target_ip` that names another host than the url is refused, not ignored
`port`	int	required for `tcp`, `dns`, `ldap`, `mssql`, `smb`	`1..65535` (also `http`, from the url or set directly)
`path`, `tls`, `server_name`, `expected_status`	mixed	`http`	`path` is the request target sent exactly as written (a query allowed; no `#fragment`, no space in the query, no path that mixes percent-escapes with characters needing encoding); `expected_status` must be an HTTP status code, `200..599`; `server_name` must be a valid DNS name and sets both SNI and the Host header, and is recorded in the result as `server_name`
`tls_skip_verify`, `insecure_skip_verify`	bool	rejected	any true value fails validation
`domain`, `transport`	string	`dns`	`transport` is `udp` (default) or `tcp`
`record_type`	string	optional (`dns`)	`a` (default: an A or AAAA answer) or `srv`; `srv` needs `domain`
`base_dn`	string	optional (`ldap`)	an RFC 4514 DN of at most 1024 bytes, `auto`, or omitted
`naming_context`	string	optional (`ldap`)	an RFC 4514 DN (at most 1024 bytes) the Root DSE must advertise; not with `base_dn`
`command`, `expected_output`, `require_qga`	mixed	`qga`	`command` limited to `hostname`, `guest-get-host-name`, `ping`, `guest-ping`, `systemctl is-active <service>`, or `sc query <service>` (Windows); `expected_output` is refused on `ping` and `guest-ping`, whose reply has no output to match; `require_qga` is recorded in the result

A field only some probe types read is refused on any other type, never ignored: a tcp probe that carries tls: true or expected_output would make a plain TCP connection while the plan reads as if it checked more. The aliases (tcp_handshake, http_get, qga_ping, qmp_status) are held to exactly the rules of their types, including the qga command list, whether a plan is loaded from a file or built in code. This table is what each type reads:

Probe type	Fields it reads besides `type`, `timeout_sec`, `retries` and `description`
`tcp`	`target_ip`, `port`
`http`	`target_ip`, `port`, `url`, `path`, `tls`, `server_name`, `expected_status`
`dns`	`target_ip`, `port`, `path`, `domain`, `transport`, `record_type`
`ldap`	`target_ip`, `port`, `base_dn`, `naming_context`
`mssql`	`target_ip`, `port`
`smb`	`target_ip`, `port`
`qga`	`command`, `expected_output`, `require_qga`
`qmp`	none
`screendump`	none

On a dns probe, path is read only as the domain to query when domain is left out. tls_skip_verify and insecure_skip_verify are refused on every type.

Plan reference

1. Plan layout

Editor completion and checks

2. Configuration field notes

compliance

admission_control

network

tiers and the dependency order