Proxmox RCU Stall errors

This guide provides instructions on how to troubleshoot and resolve Proxmox RCU Stall errors. RCU (Read-Copy-Update) stall warnings occur when a virtual machine’s compute threads are unexpectedly paused, triggering the guest kernel’s internal watchdog. This can lead to system instability, network drops, and Out-Of-Memory crashes. The steps below systematically eliminate the most common causes of these micro-stalls, ranging from host resource contention to deep CPU sleep states and Non-Uniform Memory Access (NUMA) latency.

1. Eliminate CPU Oversubscription

The most common cause of RCU stalls is allocating more total virtual CPUs (vCPUs) to running VMs than you have physical threads on your host processor. When VMs have to compete for processing time, the host scheduler may pause them, triggering the guest kernel’s warning.

• Review your VM configurations and reduce the core count on your VMs. Start with 1-2 cores per VM and only scale up if monitoring shows active utilization.
• Check your Proxmox host’s CPU graphs for high IO delay (IOwait). If the host CPU is waiting on slow storage (HDDs, a loaded NAS, etc.), it halts the vCPU.

2. Change the Host CPU Governor to “Performance”

By default, Debian/Proxmox may set the CPU frequency governor to a power-saving mode. When a VM suddenly requires CPU cycles, the physical processor takes a fraction of a second to ramp up its clock speed. This delay can be long enough to trigger an RCU stall in the guest.

On your Proxmox host, check the current governor:

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

If it returns powersave or ondemand, switch it to performance by installing and configuring cpufrequtils:

apt install cpufrequtils
echo 'GOVERNOR="performance"' > /etc/default/cpufrequtils
systemctl restart cpufrequtils

3. Enable NUMA for Multi-Socket Servers

If your Proxmox server has multiple physical processors (e.g., dual Intel Xeons or AMD EPYCs), it uses Non-Uniform Memory Access (NUMA). Each processor has its own bank of RAM. If a VM’s memory is on CPU 0 but its compute threads are scheduled on CPU 1, the cross-socket communication can cause latency spikes (micro-stalls) that trigger RCU errors.

1. In the Proxmox web interface, select the affected VM.
2. Go to Hardware -> Processors.
3. Check the NUMA box.
4. Completely shut down and start the VM to apply the change.

To verify NUMA placement, find the process ID of the VM (replace 100 with the VM ID):

cat /var/run/qemu-server/100.pid

Then check the memory allocation for that PID (replace 12345 with the PID):

numastat -p 12345

You want to see the vast majority of memory allocated to a single node.

4. Distribute Network Interrupts (SoftIRQ)

If your log mentions a massive number of Software Interrupts (e.g., softirq=1612336/1612337), the VM’s VirtIO network adapter may be overwhelmed, hammering a single core until it stalls.

1. Go to the VM’s Hardware tab.
2. Edit the Network Device.
3. Set the Multiqueue value to match the number of vCPUs assigned to the VM.
4. Completely shut down and start the VM.

5. Disable Deep Hardware C-States

Modern processors drop idle cores into deep sleep states (C6 or C7) to save power. The time it takes to wake a core from deep sleep can exceed the guest kernel’s RCU threshold. You can prevent the host kernel from allowing deep sleep by modifying the boot parameters.

First, determine which bootloader Proxmox is using:

proxmox-boot-tool status

If using systemd-boot:

1. Open the kernel cmdline file:
   “`bash
   nano /etc/kernel/cmdline

2. Append `intel_idle.max_cstate=1 processor.max_cstate=1` to the end of the existing line (ensure it remains a single line).
3. Refresh the boot tool:
```bash
proxmox-boot-tool refresh

4. Reboot the Proxmox host.

If using GRUB:

1. Open the GRUB configuration file:
   “`bash
   nano /etc/default/grub

2. Add `intel_idle.max_cstate=1 processor.max_cstate=1` to the `GRUB_CMDLINE_LINUX_DEFAULT` line.
3. Update GRUB:
```bash
update-grub

4. Reboot the Proxmox host.

6. Offload Storage I/O

Heavy storage operations (like Proxmox backups, snapshots, or intensive guest logging) can temporarily lock the QEMU main thread.

1. Go to the VM’s Hardware tab in Proxmox.
2. Ensure your SCSI Controller is set to VirtIO SCSI single.
3. Edit the Hard Disk and check the iothread box.
4. Completely shut down and start the VM.