Proxmox VE 9 replaces HA groups with flexible placement rules

Proxmox VE 9 has updated its high availability architecture by replacing legacy HA groups with more granular HA rules. This change affects how administrators define where virtual machines and containers run during normal operation and after a node failure. The update requires careful configuration of node affinity and resource placement to ensure reliable failover without manual intervention.

What happened

The latest version of Proxmox VE shifts the mechanism for controlling guest placement from simple HA groups to a rule-based system stored in /etc/pve/ha/rules.cfg. In previous releases, administrators used groups to express preferences for which nodes should host specific guests. Proxmox VE 9 deprecates these groups in favor of explicit node affinity and resource affinity rules. Existing configurations are automatically migrated to node affinity rules once every node in the cluster runs version 9, but administrators must verify the resulting logic to avoid unexpected behavior.

This structural change is part of a broader emphasis on robust high availability setups that rely on proper quorum and fencing. The system uses two primary daemons: the local resource manager (pve-ha-lrm) and the cluster resource manager (pve-ha-crm). These components coordinate with a watchdog timer to ensure that if a node fails or loses network connectivity, it is forcibly reset before its guests are restarted elsewhere. This prevents data corruption on shared storage by ensuring no two nodes write to the same disk simultaneously.

Key details

• HA Rules replace Groups: Node affinity rules define where a guest can run, while resource affinity rules keep guests together or apart. Rules are not strict by default unless explicitly configured.
• Watchdog Fencing: Proxmox VE uses a watchdog timer to fence failed nodes. If a node loses quorum, it cannot reset the timer, causing a hardware or soft reset to protect shared storage integrity.
• Quorum Requirements: Reliable high availability requires at least three cluster nodes or two nodes plus a QDevice. Two-node clusters without a QDevice risk split-brain scenarios where no node can safely take over.
• Shutdown Policies: Administrators can choose between conditional, migrate, failover, or freeze policies for planned maintenance. The default conditional policy stops guests on shutdown, while migrate attempts live migration first.
• Maintenance Mode: A dedicated maintenance mode command migrates HA guests away from a target node and keeps them off until the mode is disabled, simplifying hardware repairs or updates.
• Testing Requirement: The author emphasizes pulling the power cord on a test node to verify failover, as clean shutdowns do not trigger the same fencing and recovery mechanisms as actual hardware failures.

Background

High availability in virtualization environments relies on the concept of quorum, which ensures that a majority of nodes agree on the cluster state. Without quorum, a cluster cannot distinguish between a node failure and a network partition, leading to potential data corruption if multiple nodes try to act as the primary. Proxmox VE addresses this by requiring an odd number of voting members or a tie-breaking QDevice.

Fencing is the process of isolating a failed node to prevent it from accessing shared resources. In Proxmox VE, this is achieved through watchdog timers. If a node becomes unresponsive or loses contact with the majority of the cluster, its watchdog timer expires, forcing a reboot. This hard reset ensures that any locks held by the failed node are released, allowing the cluster resource manager to safely restart the affected guests on healthy nodes.

Why it matters

For teams running self-hosted infrastructure, the shift to HA rules offers greater control over resource distribution but introduces complexity in configuration. Misconfigured affinity rules can prevent guests from starting during a failure if the preferred nodes are unavailable. Understanding the difference between strict and non-strict affinity is critical to ensuring that essential services remain online even when preferred hardware is offline.

The reliance on watchdogs and quorum means that hardware selection and network reliability are more important than ever. Using the default software watchdog (softdog) is convenient but less reliable than a dedicated hardware watchdog module. Teams must ensure their server hardware supports and enables these modules to guarantee that fencing occurs promptly during real-world failures, such as power loss or kernel panics.

Furthermore, the distinction between high availability and backups remains a common pitfall. HA ensures service continuity by restarting guests on other nodes, but it does not protect against data corruption or logical errors within the guest itself. Administrators must maintain separate backup strategies and ensure that shared storage solutions, such as Ceph or ZFS replication, are properly configured to minimize data loss windows during failover events.

What you can do

• Verify Quorum Setup: Ensure your cluster has at least three nodes or two nodes with a QDevice to maintain quorum during a single node failure.
• Review HA Rules: Check /etc/pve/ha/rules.cfg after upgrading to Proxmox VE 9 to confirm that migrated node affinity rules match your intended placement strategy.
• Configure Hardware Watchdogs: Enable hardware watchdog modules in /etc/default/pve-ha-manager instead of relying solely on the software softdog for more reliable fencing.
• Test Failover Safely: Perform a controlled test by adding a disposable VM to HA and physically cutting power to its host node to observe the recovery process and timing.
• Set Shutdown Policy: Explicitly set the shutdown policy in datacenter.cfg to migrate if you prefer live migration during planned reboots, rather than the default stop-and-restart behavior.
• Use Maintenance Mode: Utilize the ha-manager crm-command node-maintenance feature when performing work on a specific node to automatically migrate guests away and prevent them from returning until the work is complete.