Skip to content

Guest network recovery

A static Linux guest often ties its production network connection to an interface name or a PCI path, and that tie can break in an isolated recovery VM even when the MAC address is kept. For a VM with Stage 2 wire probes, CertiStack therefore writes one temporary connection profile into the VM's disposable copy-on-write (COW) overlays before it boots. The source VM, the PBS snapshot and the read-only mapped image are never changed.

This page says which guest files recovery reads, what it writes, what it refuses and how to turn it off. The plan keys are in the plan reference. network_recovery is a key of a VM, written under that VM in tiers[].vms[] (next to its probes), not at the top of the plan. Most plans need none of them, only an optional address, a legacy interface name or failure diagnostics:

# Under a VM in tiers[].vms[]:
network_recovery:
  mode: auto
  # Optional when all Stage 2 probes already target one address.
  sandbox_ip: "192.0.2.10"
  # Optional compatibility fallback for a legacy guest profile bound to a
  # predictable Linux device name; writes a MAC-bound COW-only .link file.
  interface_name: "eth0"
  # Optional: keep guest-side diagnostics when a wire probe fails.
  capture_diagnostics: true

A VM that omits network_recovery runs in auto mode: CertiStack infers the one guest address from that VM's Stage 2 probe targets and detects the network backend. A VM whose wire probes target more than one guest address fails before recovery, rather than guessing which identity to rewrite; set sandbox_ip. Recovery assigns only the isolated address. It creates no gateway, default route, DNS setting, SNAT or physical uplink. The report times it apart from mapping and boot (see recovery time).

What it needs

  • guestfish (package libguestfs-tools) on the PVE node that runs the worker. A node without it fails preflight for a plan that needs network recovery; mode: preserve does not need it. Recovery never falls back to changing a source guest or attaching it to production networking.
  • A guest the restored disks explain. Recovery opens the overlays of every restored disk together, so a guest whose root file system is on a second disk, or whose LVM volume group spans several, is inspected whole. The profile is written into whichever overlay holds /etc. Recovery fails before boot when a disk the root file system or a volume group needs is one the plan leaves out, or when a data disk holds a second operating system (guestfish -i takes one). Opening every disk reads more of the backup than the boot disk alone, and the reads count in the recovery timing.
  • No encrypted disks. A restored disk that holds a LUKS volume would make guestfish ask for a passphrase, which CertiStack neither has nor asks for. The run fails before boot, in auto mode, with an error that names LUKS (or BitLocker) and the way out: mode: preserve, under which the guest keeps its own network configuration and boots only if it unlocks unattended (for example from a key file in its initramfs). You can also use probes that need no guest network (qga), or leave out an encrypted data disk the guest does not need.

Which backend is used

auto detects legacy RHEL and Rocky ifcfg, NetworkManager, Netplan, systemd-networkd, or Debian-style ifupdown, and binds the profile to the preserved MAC. You can also name the backend with mode: ifcfg, networkmanager, netplan, systemd-networkd or ifupdown. auto follows the backend that is in charge of the guest's network, not the configuration directories its image happens to keep, and writes a profile only where the guest will read it:

  • A masked NetworkManager.service (its unit in /etc/systemd/system is a link to /dev/null, or empty) owns nothing. A guest that keeps /etc/NetworkManager while systemd-networkd runs is adapted through Netplan or networkd. If every service that could read a profile is masked, recovery fails before boot and names them.
  • When Netplan renders a definition that can match the NIC with networkd, that definition owns the NIC even with NetworkManager enabled beside it. With NetworkManager disabled, an enabled networkd or ifupdown configures the guest.
  • When NetworkManager and networkd are both enabled and networkd files can match the NIC, CertiStack cannot tell which one configures it and asks for an explicit mode.
  • An explicit mode is honored, except that it is refused, with the mode to use instead, when the service that would read its profile is masked, or (networkmanager) when Netplan hands the NIC to networkd.

CertiStack decides this from the guest's files. It does not boot the guest to observe its network, and a tool that rewrites the network configuration at boot (cloud-init on a changed instance) is not modeled.

ifcfg and NetworkManager

On EL7, whose NetworkManager ignores keyfile profiles unless configured to load them, auto uses a MAC-bound ifcfg profile. When a NetworkManager guest keeps an ifcfg profile bound to the restored NIC's MAC (HWADDR), auto rewrites that profile in place, which keeps its SELinux label, and gives the NIC the profile's own DEVICE name (the part of the file name after ifcfg- only when the profile has no DEVICE).

Files that NetworkManager's ifcfg plugin ignores are never taken for that profile: names ending in .bak, ~, .orig, .rej, .rpmnew, .rpmorig, .rpmsave, .augnew, .augsave or .augtmp (in any case), and rpm's ; plus eight hexadecimal digits. A package upgrade leaves ifcfg-eth0.rpmsave beside an edited ifcfg-eth0. An interface_name that would make the profile's own file name one of them is refused. When more than one loadable profile is bound to the MAC, CertiStack asks for interface_name instead of rewriting whichever the directory lists first.

CertiStack writes a NetworkManager profile where the guest's NetworkManager reads keyfiles. It evaluates the guest's configuration as NetworkManager.conf(5) describes it, later values overriding earlier ones: the snippets of /usr/lib/NetworkManager/conf.d, then those of /run/NetworkManager/conf.d, then /etc/NetworkManager/NetworkManager.conf, then the snippets of /etc/NetworkManager/conf.d, and last the internal state file /var/lib/NetworkManager/NetworkManager-intern.conf. Snippets are the .conf files, in file-name order; a file of the same name in /etc (or /run) shadows the one below it, a link to /dev/null masks a snippet, and [.config] enable=false switches one off. From the result it takes [keyfile] path (the conventional /etc/NetworkManager/system-connections when it is not set) as the profile directory, and [main] plugins, with its += and -= forms, for the EL7 check.

A configuration it cannot evaluate offline is refused before anything is written, naming the file: a file that is not valid key-file syntax, a [keyfile] path that is relative, holds .. or a backslash, or is empty, a snippet that sets either key but is enabled only under a version or environment condition, a configuration directory that is a link, a snippet that is a link to anything but /dev/null, and a configured keyfile directory that does not exist. NetworkManager's command-line overrides (--config, --config-dir, --intern-config) and distribution patches are not in the image and are not modelled. A guest that NetworkManager does not configure (Netplan, networkd or ifupdown) is not refused for a file it never reads.

ifupdown

ifupdown (networking.service and /etc/network/interfaces) cannot match a MAC. An OpenRC guest such as Alpine Linux is recognised too, when no systemd backend configures it, its networking init script is linked into the boot, default or sysinit runlevel under /etc/runlevels, and /etc/network/interfaces exists. When the guest has cloud-init configuration (/etc/cloud/cloud.cfg.d), recovery also writes 99-certistack-recovery-network.cfg there (network: {config: disabled}): some images, Alpine among them, let cloud-init regenerate /etc/network/interfaces at every boot, and without its seed it would fall back to DHCP and replace the recovery stanza. Recovery replaces the overlay's /etc/network/interfaces with a static stanza for the guest's configured interface and keeps the original beside it as interfaces.certistack-original.

It finds the guest's interface names in that file and in the files it includes with source (shell wildcards, paths relative to the including file) and source-directory (files whose names are letters, digits, _ and -), following nested includes within bounds: 64 files, 8 levels, 1 MiB of text. /etc/network/interfaces.d is read only when the main file includes it, so a stale file there is not taken for an interface. An include that needs a shell to evaluate (variables, command substitution, quoting, ~) is refused, naming it: set interface_name or use preserve. The rules are those of Debian's ifupdown (interfaces(5)); how ifupdown2 and ifupdown-ng filter a source-directory is not modelled separately.

Bridges, bonds, VLANs and tunnels are set aside in favor of the physical interface under them. When the guest names several physical interfaces and the sandbox VM has one NIC (for example cloud-init's enp6s18 beside an image's stale eth0), each name gets an allow-hotplug stanza and only the one that exists comes up.

With several NICs (attach_all_nics), recovery asks for interface_name, and the replacement file keeps the guest's own stanzas for the other interfaces: every stanza of the main file and of the included files that is not the recovered NIC's (or lo's) is copied as written, the auto and allow-* lines lose only the recovered NIC's name, and the file has no source line, so the fragments, left unread, cannot bring the recovered NIC's own stanza back. A stanza built on the recovered NIC (a bridge, bond or VLAN over it, bridge_ports all), a mapping, rename or template stanza, and an option line outside any stanza are refused before anything is written, naming attach_all_nics: false and preserve as the way out.

Netplan and systemd-networkd: winning against the guest's own files

systemd-networkd applies the first .network file, in file-name order, whose [Match] selects the link. Netplan writes one file per definition ID (10-netplan-<id>.network), so the definition's ID decides, not the name of its YAML file. CertiStack reads the guest's Netplan definitions (merged across /lib/netplan and /etc/netplan as Netplan merges them) and the systemd network directories, and names its recovery definition the first of certistack-recovery, Certistack-recovery, 0-certistack-recovery and -certistack-recovery that sorts before every definition or file that can match the recovered NIC. The .network file of the systemd-networkd mode and the interface_name .link file are named the same way. The guest's own files are never changed. Recovery fails before anything is written, naming the cause, when a competing Netplan definition is rendered by NetworkManager (which applies its own connection whatever networkd reads; use mode: networkmanager), when a Netplan file cannot be read, or when nothing sorts first.

The interface_name .link file has to win the same way, or the NIC gets another name than the profile uses. udev applies the first matching .link file in file-name order across /etc/systemd/network, /run/systemd/network, /usr/local/lib/systemd/network, /usr/lib/systemd/network and /lib/systemd/network; a file hides one of the same name in a later directory, and a link to /dev/null hides it altogether. CertiStack reads the rule names in all five and names its rule the first of 00-certistack-recovery.link, 0-certistack-recovery.link and -certistack-recovery.link that sorts before every rule that can match the recovered NIC. A rule with a drop-in directory, or one that is not a plain file, counts as able to match. When no name sorts first, recovery stops before writing and names the rule; mode: preserve is the way out. Rules inside the guest's initramfs are not visible offline and are not modeled.

The .network scan reads the same five directories under systemd/network with the same shadowing and masks. MAC matches and, when interface_name is given, name patterns exclude profiles for other NICs; unknown names remain possible matches. The NetworkManager ownership check reads /etc/systemd/network only, so shipped container profiles alone do not make ownership ambiguous. Two limits matter only for a hostile image: NetworkManager snippet names containing a newline cannot be distinguished in guestfish's line-oriented listing and may be skipped, and a guest whose /etc/systemd/network or NetworkManager keyfile directory is itself a symbolic link is treated as having no such directory.

How the overlay is written

Everything recovery adds to the overlay (the profile, the interface .link file, the Netplan wait-online drop-in, the diagnostic scripts and units, the firewalld zone, the ifupdown replacement) goes to a fixed path, and guestfish writes to a path by following a symbolic link there. So each path is looked at before anything is written:

  • A regular file is written in place, which keeps its SELinux label. Where nothing exists, the file is created.
  • A symbolic link, a directory, a device, a pipe, a socket or a file with other hard links is refused with its path named, because the write would land in an unrelated file of the guest.
  • A profile copied to keep the label is chosen among the guest's regular files only. The written file is set to root:root with the profile's mode (NetworkManager ignores a keyfile that anyone but root can read or write), then read back, and the run stops if what it finds is not a regular file owned by root with that mode.
  • The two enablement links of capture_diagnostics are created only where nothing is: a link that already points at the wanted unit is left as it is (and is still listed in the VM record), and any other entry there, a link to something else, a regular file or a directory, is refused, not replaced.

mode: preserve is the way out of a refusal for an image that cannot be repaired. The VM record lists the chosen strategy, the address and every guest path recovery wrote (network_recovery_files): the profile, the .link file and the Netplan drop-in when there are any and, with capture_diagnostics, the diagnostic script, its two units and the NetworkManager dispatcher copy, and the two enablement links, which are listed as path -> target.

firewalld

When a guest has firewalld installed, the same overlay receives source-restricted rules in the zones that can handle the probe, so the sandbox probe address may reach the configured Stage 2 probe ports. That makes an otherwise correct restored service testable even when its own firewall would refuse the probe, without opening a production firewall or changing the backup. The change disappears with the overlay.

firewalld hands a packet to the zone bound to its source address, else to the zone bound to its interface, else to the default zone, and NetworkManager may move the NIC between zones when it activates the profile. The rules therefore go to the guest's effective default zone (the DefaultZone of firewalld.conf, or, as firewalld does when that names no defined zone, public, then external, then block) and to every administrator zone that binds an interface or a source address that can hold the probe address (a MAC-address or ipset binding cannot be ruled out offline). Each zone file is read as XML and the rules are added as the last children of its one zone element. Everything the zone already allowed is kept byte for byte, and the result is read again to check that only the probe rules were added. A vendor zone is copied under /etc/firewalld/zones first.

Recovery fails before boot, naming the zone, instead of writing rules that would not take effect: when a zone file is not a single well-formed zone document (firewalld drops its whole /etc/firewalld configuration for the stock one when a zone fails to load), when a zone directory would be combined into one zone, or when a zone has a rich rule that certainly drops or rejects the probe's traffic to a declared port at a priority firewalld evaluates before accept rules. Rules whose match cannot be evaluated offline (services, ipsets, MAC addresses, destination addresses), firewalld policy objects and run-time changes made inside the guest are not evaluated. A wire probe that is still refused fails and reports it: the added rules are configuration, and the probe is what shows reachability. CertiStack adds no rule for guest-to-guest traffic, so restored guests on the same VNet reach each other as far as their own firewalls allow. Routed multi-VLAN topologies are outside the supported boundary.

Diagnostics: capture_diagnostics

With capture_diagnostics: true, a failed wire probe makes CertiStack record the best safe guest-side evidence available (interfaces, routes, listening sockets and relevant service state) in the report. Guest-agent diagnostics are optional: an unavailable or restricted agent is reported as such and does not turn a passing wire probe into a failure.

When the guest agent cannot run the diagnostic, recovery installs a shell script in the overlay that runs at boot, on each NetworkManager device event and once boot has settled. Those can overlap, so no capture replaces another: each is published whole, under a name stamped with when it began, in /var/lib/certistack. CertiStack reads the one that began last, and the newest eight stay in the overlay. The script writes as root, so it requires that directory to be a real directory (not a link), makes it root-only (0700), and writes through an exclusively created temporary file, so a link planted in the guest cannot redirect the write. It removes abandoned .capture.* temporary files at its next invocation, keeping those of a capture that is still running. Every section is cut at a fixed number of bytes and the file at 32 KiB, and at most 16 KiB are read back (a cut capture ends with [truncated]): a verbose guest costs the evidence beyond the limit, never the run.

The overlay is read during teardown, after the sandbox VM is confirmed stopped and before PVE destroys it. QEMU is no longer writing to the overlay then, and PVE's destroy removes it, because it is a volume of the sandbox VM. The read has a 3-minute deadline of its own, and the destroy waits for it. When the VM cannot be confirmed stopped the overlay is not read, and the diagnostic in the report says it was not read and why. An overlay that is already gone, or that cannot be read, is said in the diagnostic and is not a cleanup failure.

interface_name

Use interface_name only for a confirmed legacy profile that is tied to an interface name (for example ifcfg-eth0). CertiStack adds a MAC-specific .link file beside the temporary recovery profile so the guest sees that name in the sandbox. It never renames a source interface. The name must be one Linux accepts and CertiStack can write into an ifupdown stanza, an ifcfg file name and a .link file: 1 to 15 characters from ASCII letters, digits, _, - and ., and not lo, . or .. (the kernel rejects . and ..). A plan with any other name is refused when it is validated.

More than one NIC

Only net0 is attached by default. Set attach_all_nics: true to attach the source's other NICs (net1 and up) as well. Each keeps its MAC and model and is attached, like net0, only to the isolated VNet with the PVE firewall on; their bridges, VLAN tags and firewall settings are never replayed. Use it when a service binds to a secondary interface's address and would not start without it. The report lists attached NICs under restored_hardware.additional_nics and the ones left out under omitted_nics. The rules for their MAC addresses are in NIC MAC addresses.

Only net0 gets a recovery address. The other NICs keep the guest's own configuration (for an ifupdown guest, whose recovery file replaces /etc/network/interfaces, their stanzas are copied into it), and a guest that waits for DHCP on them boots more slowly, because the sandbox has no DHCP. They bring up their production static addresses on the sandbox VNet. An address outside network.ip_range is not one a probe can reach. An address inside the range is reachable, so it must be unique in the sandbox: not probe_ip, not another VM's sandbox_ip or probe target, and not the subnet's network or broadcast address. Otherwise it could answer another VM's probe or clash with probe_ip. Validation checks only the addresses a plan names as probe targets (below); it cannot see an in-range address of a secondary NIC that no probe names, so for such an address check the uniqueness yourself, or leave attach_all_nics off.

To probe a service on another NIC's address, set sandbox_ip to net0's address and point the probe at the other NIC's address. That address is the guest's own, so it must be inside network.ip_range to be reachable, and unique in the plan: validation refuses a target that equals probe_ip, the subnet's network or broadcast address, or an address of another VM. Whether the guest's firewall admits the probe on that interface depends on the guest's configuration of it.

Windows and other guests it cannot adapt

Recovery rewrites Linux network configuration. It cannot adapt a Windows guest, or any guest whose network configuration it does not recognise, and a VM with wire probes and the default mode then fails with an unsupported guest network layout error that names network_recovery.mode: preserve. For such a guest:

  • set network_recovery.mode: preserve, so the guest keeps its own address on the isolated VNet;
  • make network.ip_range the subnet the guest already uses, and each wire probe's target_ip that guest's own address (see Supported versions and limits), provided the PVE node that runs the test has no address and no route in that subnet: the sandbox is a disconnected layer-2 domain, but while a run lasts the node routes the whole network.ip_range into it, so a node with a leg in the guest's subnet would send its own traffic for that subnet into the sandbox. plan, check and run refuse such a range (Issue 40). When the node has a leg in the guest's subnet (a guest that lives on the node's management network, or on any bridge that carries a host address), run the test on a node of the cluster that has none, or re-address the guest to a subnet of its own: a guest that keeps its production address cannot be given a different one by CertiStack, because preserve is the mode for guests whose address it cannot change. For a guest that recovery can adapt (the default mode), none of this applies: it receives a sandbox address, so choose any range your node does not use;
  • do not set sandbox_ip, which preserve forbids.

Appliance images belong here too, even when they are Linux or have a guest agent. In the default mode they fail before boot, and the message names preserve:

Guest What the default mode reports
pfSense, OPNsense, MikroTik RouterOS, VyOS (not Linux with a recognised network layout: their network lives in their own configuration file or database) no operating system was found on this disk, or unsupported guest network layout after a libguestfs error such as inspect_os: parted exited with status 1: Error: Invalid partition table - recursive partition (FreeBSD disks); either way with the preserve remedy
Home Assistant OS (a read-only root file system, with the writable state on separate partitions) unsupported guest network layout: the guest's file system is read-only in the image after upload: /etc/NetworkManager/system-connections/...: Read-only file system, with the preserve remedy

Whichever of these you see on an appliance, the run has stopped before booting the VM and cleaned up; set network_recovery.mode: preserve and probe it with qga, or with a wire probe at the address the appliance already holds.

Guest-agent probes (qga with sc query <service>) need no guest network at all. The Windows domain example uses both. use_backup_config: true remains the recommended path (see backup configuration inheritance).