9:50 a.m.
I have a very strange issue that has plagued our KVM-based hypervisor
infrastructure.
Environment:
- All EL7-based libvirt releases since EL7.0
- Fully patched EL7-based KVM guest on a fully patched EL7-based KVM host.
- Secondary disk attached via 'virsh attach-device guest
/vm/guest-with-secondary-disk-01/disk2.xml --persistent' (note --persistent
flag here)
Problem Description:
- Guest becomes unresponsive/hangs due to issue on guest. Typically the
scenario is that PID1 hangs or goes into a bad state due to a misbehaving
application.
- Before virsh destroy, 'virsh dumpxml guest | grep vd[ab]' returns the
expected filesystems.
- Guest is destroyed from the hypervisor via 'virsh destroy guest'.
- With guest still powered off, 'virsh dumpxml guest | grep vd[ab]' only
shows 'vda'. 'vdb' now unattached.
- To fix, requires re-run of 'virsh attach-device guest
guest-with-secondary-disk-01/disk2.xml --persistent'
Expected Behavior:
- 'virsh destroy guest' does not drop the secondary disk attached to the
guest if the node is in a hung state.
Other notes:
- If guest is not in a hung state, 'virsh destroy' does not drop the
secondary disk.
- 'virsh reset guest' does not exhibit the same behavior when guest is hung
- vdb remains attached.
- No logs produced in /var/log/libvirt indicating an issue, even with debug
logging enabled.
Is there any obvious I'm missing? Is this expected behavior?
Guest XML via 'virsh dumpxml guest':
<domain type='kvm' id='17'>
<name>guest-with-secondary-disk-01</name>
<uuid>ab578b51-3afd-4d01-86ab-a16b1b953919</uuid>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>1</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64'
machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source
file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01.img'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source
file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x08'
function='0x0'/>
</disk>
<controller type='usb' index='0' model='piix3-uhci'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x06'
function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='01:02:03:99:99:99'/>
<source bridge='bridge_99'/>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='eth0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x0'/>
</interface>
<channel type='unix'>
<source mode='bind'
path='/var/lib/libvirt/qemu/guest-with-secondary-disk-01.agent'/>
<target type='virtio' name='org.qemu.guest_agent.0'
state='connected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0'
port='1'/>
</channel>
<input type='mouse' bus='ps2'>
<alias name='input0'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input1'/>
</input>
<graphics type='vnc' port='5900' autoport='yes'
listen='127.0.0.1'
keymap='en-us'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1'
primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
</memballoon>
<rng model='virtio'>
<backend model='random'>/dev/random</backend>
<alias name='rng0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x07'
function='0x0'/>
</rng>
</devices>
<seclabel type='dynamic' model='selinux' relabel='yes'>
<label>system_u:system_r:svirt_t:s0:c259,c495</label>
<imagelabel>system_u:object_r:svirt_image_t:s0:c259,c495</imagelabel>
</seclabel>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+107:+107</label>
<imagelabel>+107:+107</imagelabel>
</seclabel>
</domain>
Secondary disk XML:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source
file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/>
<target dev='vdb'/>
</disk>