Awful slowness with Dell R620 (20 cores), ACPI 4, and Debian Wheezy (Linux 3.2.0)

Summary

This problem has not been adequately described elsewhere, as far as I can tell. I'll try to keep it brief. Some symptoms and a more prosaic description of the problem is included below, but it really just states the same.

The problem

The system is performing horribly, and watchdog and power_saving threads try to use all available CPU time.

Affected systems

EDIT October 25, 2017: Kernel version has no apparent significant effect on the issue. The problem appears to affect all kernel versions for any stable Debian.

1. Linux kernel older than 3.5.0 (e.g. Debian Squeeze or Wheezy, Centos 6.2, RHEL 6)
2. EDIT October 25, 2017: Any Debian up to and including Stretch
3. ACPI 4 compliant hardware (e.g. Dell 12g systems, R620, R720, ...)
4. Logical processors enabled (hyperthreading, virtual CPUs)
5. Power-saving enabled
6. System is not busy, but not completely idle

Mitigation

These are temporary solutions before you can reboot with a more recent kernel.

EDIT October 25, 2017: These are the first steps to mitigate the problem while avoiding a reboot.

1. rmmod acpi_pad (this takes a while)
2. echo 0 > /proc/sys/kernel/nmi_watchdog

Solution

EDIT October 25, 2017: Upgrading Debian's kernel has no significant effect, as the problem persists. Only blacklisting the module acpi_pad works.

1. rmmod acpi_pad (this takes a while)
2. echo "blacklist acpi_pad > /etc/modprobe.d/acpi_pad.conf"

Upgrade the kernel to something recent, e.g. by compiling the current long term release yourself.

Debian Wheezy users may use the most recent kernel from wheezy-backports, which as of 2015-05-21 is stuck at 3.16. Please note that recent Linux kernels are compressed with xz. If your system is a virtual machine, ensure that the host system supports LZMA compressed kernels!

Symptoms

Maybe you got an alert about 100+ load, or a user report about system slowness. You look at the system, which is indeed responding slowly even to basic commands, and see watchdog and/or power_saving threads eat nearly all your processing capability (100% or more CPU usage).

Here's an example of what top(1) may show you. On some occasions, the power_saving processes may be dominating.

PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND            
   64 root      rt   0     0    0    0 R   331  0.0  52:22.64 watchdog/14        
   72 root      rt   0     0    0    0 R   324  0.0  51:35.85 watchdog/16        
   12 root      rt   0     0    0    0 S   324  0.0  53:32.19 watchdog/1         
   79 root      rt   0     0    0    0 R   322  0.0  50:46.77 watchdog/17        
   92 root      rt   0     0    0    0 R   315  0.0  50:24.58 watchdog/19        
  100 root      rt   0     0    0    0 R   309  0.0  54:09.71 watchdog/21        
  191 root      rt   0     0    0    0 R   301  0.0  62:52.38 watchdog/39        
  183 root      rt   0     0    0    0 R   299  0.0  66:01.97 watchdog/37        
  116 root      rt   0     0    0    0 S   296  0.0  52:55.74 watchdog/25        
  141 root      rt   0     0    0    0 S   295  0.0  54:26.20 watchdog/29        
  121 root      rt   0     0    0    0 S   295  0.0  52:59.55 watchdog/26        
  133 root      rt   0     0    0    0 S   289  0.0  53:37.40 watchdog/27        
  137 root      rt   0     0    0    0 S   286  0.0  53:33.76 watchdog/28        
  145 root      rt   0     0    0    0 S   279  0.0  57:03.61 watchdog/30        
  161 root      rt   0     0    0    0 S   268  0.0  60:25.99 watchdog/34        
   24 root      rt   0     0    0    0 R   258  0.0  51:34.17 watchdog/4         
    7 root      rt   0     0    0    0 R   251  0.0  53:31.10 watchdog/0         
   40 root      rt   0     0    0    0 R   218  0.0  50:17.33 watchdog/8         
   44 root      rt   0     0    0    0 R   214  0.0  40:28.09 watchdog/9         
   88 root      rt   0     0    0    0 R   214  0.0  50:43.48 watchdog/18        
  149 root      rt   0     0    0    0 S   197  0.0  61:30.01 watchdog/31        
   68 root      rt   0     0    0    0 R   195  0.0  50:57.25 watchdog/15        
  157 root      rt   0     0    0    0 S   139  0.0  57:46.94 watchdog/33        
   52 root      rt   0     0    0    0 S   137  0.0  53:56.98 watchdog/11        
44382 root      -2   0     0    0    0 R    99  0.0  16:33.87 power_saving/7     
44396 root      -2   0     0    0    0 R    97  0.0  16:08.18 power_saving/21    
44400 root      -2   0     0    0    0 R    97  0.0  18:03.40 power_saving/25    
44401 root      -2   0     0    0    0 D    94  0.0  17:28.80 power_saving/26    

Likely cause

Linux kernels before 3.5 have a bug in ACPI 4 processor idling (https://bugzilla.kernel.org/show_bug.cgi?id=42981). This can cause horrible slowness, probably some kind of deadlock, when running a somewhat idle (but not totally idle or busy) system, if you have a multi-core setup, with hyperthreading (logical/virtual cores) enabled, where you also try to conserve power, and where the kernel attempts to let cores go idle.

See also: http://en.community.dell.com/support-forums/servers/f/1466/t/19456558

Useless advice

I've also found the following useless/harmful advice. The text has been paraphrased by me. My comments are in parentheses:

1. "Reboot, that solved the problem for me" (It "solves" it for a very brief period of time, until the problem reappears. Brilliant for not keeping the system up, though.)
2. "Disable logical processors in BIOS" (That takes us down from 40 to 20 logical CPUs, yay, big help. Not if you want to keep the system up for a while.)
3. "Disable power-saving, it's a server, this is a laptop feature" (I bought power-saving CPUs and memory for a reason, thank you! See also "keeping the system up".)