Hello all, I'm new to the list, and hoping someone can point me in the right direction.

I've got an Ubuntu 14 dom0 with, what I think, are excellent specs. 12 cores, hyperthreaded, VT, 64GB RAM, gobs of disk, etc... all to run 8 virtual machines (at the moment).

My problem is each of the guests - Windows, Ubuntu, FreeBSD, starts out working fine. After several minutes, the guest becomes unresponsive for a moment, and comes back. As time goes on, this happens more, and more frequently, and longer and longer until after about a day the guest is completely useless, and needs a shutdown, and start.

The dom0 is NEVER unresponsive.

I've tried to track this down through every means I have available, and have come up empty handed. Here is some of what I've done so far, please let me know what other information would help to debug this:

1. I've set guest OS processor to be default, and copied from host. Neither made any noticable difference.

2. I've switched all disks and network to be VirtIO. Big improvement over IDE, but the unresponsive problem persists.

3. Linux guests have the following in dmesg:
hrtimer: interrupt took 10109276 ns
[sched_delayed] sched: RT throttling activated

4. I found nothing suspicious in the dom0's dmesg.

5. Unresponsiveness does not correlate with disk usage.

6. Host uses 4 disks in software RAID-5 (yes I know I'm bad, but there are reasons) using BTRFS.

7. Guest disks are all raw.

Please let me know if there is some other useful information, or if you have an idea where I should look next.

Thanks!