What to expect from a run
A run restores every VM in the plan from its newest backup into an isolated sandbox, boots it, tests it, and removes it again. That takes minutes, not seconds, and most of the time goes to a few steps you can predict. This page says which, with measured figures, so a first run is not a surprise.
Where the time goes
| Step | What it depends on | What it looked like |
|---|---|---|
| Admission, sandbox network, mapping the backup, creating the overlay | Nothing you control; the node's SDN reload is the longest part | Under 10 seconds in every measurement |
| Guest network preparation | libguestfs starts a small appliance VM several times to read and rewrite the guest's network settings, inside the throwaway overlay only. Fast on a node with its own hardware virtualization, slower on a node that is itself a VM with nested virtualization turned on (a node without /dev/kvm is refused at the node preflight). A VM set to network_recovery.mode: preserve skips it. |
116 s median (90 % under 185 s) on bare-metal lab nodes; 4 m 10 s for a Rocky 9 VM and 2 m 46 s for a Debian 12 VM on a small nested single-node PVE |
| Image integrity scan | The size of the VM's disks divided by how fast the node reads them back from PBS. The scan reads every disk once, in full, and hashes it; it is what makes the report a record that every disk was read in full. | 385 s median (90 % under 669 s) across the lab's mixed VMs; 84 s for 12 GiB on the small nested node |
| Boot and registration | The hypervisor only | About 5 seconds |
| Probes | How long the guest takes to start the service. Until the probe answers, the run keeps trying within the probe's timeout_sec; a plan with no startup_grace_period_sec (what certistack init writes) starts probing at once and ends at the first success. |
A Rocky 9 VM answered SSH 34 s after boot, a Debian 12 VM 20 s. Java stacks (OpenSearch, Graylog) took several minutes in the lab. |
| Teardown | The number of VMs | About 10 seconds |
The whole run, measured by the report as the recovery time (RTO), came to:
| Measurement | RTO |
|---|---|
| One 12 GiB Rocky 9 VM, SSH probe, small nested single-node PVE (the time the quickstart quotes) | 6 m 20 s |
| That VM and a 6 GiB Debian 12 VM in one plan | 9 m 04 s |
| 392 passing single-VM runs in the lab (a mix of Linux distributions and Windows images of different sizes), median | 12 m 06 s |
| The same 392 runs, 90th percentile / largest | 21 m / 56 m |
The lab figures include a start grace of 180 s that the lab's plans set and a plan written by certistack init does not, so a plan like yours should come in a little under them for the same disks. They are measurements on one lab's bare-metal nodes, not a promise: your disk sizes and your PBS read speed decide most of it.
Estimate your own
The two terms you can estimate before the first run are the scan and the preparation:
- Scan time is the total size of the VM's disks divided by what your node reads from PBS. Time one run, then read it off the report:
certistack report show <report.json>prints "Image integrity scan" next to "Guest network preparation" under "Where the time went". - Preparation time is minutes on a nested node and about two minutes on bare metal (the lab's median was 116 s). It is per VM: in the two-VM run above the VMs were prepared one after another.
- Everything else together is usually under a minute, apart from a slow-starting service.
A plan with several VMs adds each VM's preparation and scan.
What the run prints while it works
Progress goes to standard error. A few lines are long pauses by design:
Still working: ... Detecting the guest network manager from the disposable overlay.is the guest network preparation above; it repeats about every half minute and the first run of a large VM can sit there for several minutes.full mapped-image integrity scan: N GiB verified.counts up to the size of the disk.Cleanup verifiedandJournal recovery verifiedat the end say that the run's cleanup was checked against the node's journal; they are worth the ten seconds.
Nothing else needs you. The first Ctrl-C stops the run and waits for it to clean up; a second one abandons the cleanup. If something is left behind, Recover from an interrupted run and certistack recover cover it.
When a run is slower than this
- A node that is itself a VM (a nested PVE lab, with nested virtualization turned on, which the node needs: without
/dev/kvmthe run stops at the node preflight): guest network preparation takes minutes, as in the walkthrough above. It is slow, not broken. - PBS across a slow link: the scan is bound by the read speed; the run does not shorten it, because skipping it would make the report mean less. The only shortcut is a test-only mode that the report labels and
verify-reportrefuses as evidence (configuration). - A service that starts slowly (a Java stack, a database that replays a log after a crash-consistent backup): raise that probe's
timeout_secrather than the plan's start grace, so a fast guest still ends the wait early. - A busy node: the run waits and retries while the node is above the plan's RAM or I/O wait limits (
admission_control), and says so.