[libvirt] dynamic scsi disk attach seems to be broken in qemu(-kvm)-0.11, libvirt-0.7.1

We're experiencing what we believe to be a regression in qemu-kvm-0.11 and libvirt-0.7.0, from previous versions of kvm-84 and libvirt-0.6.1: Dynamically attaching sdb (a second scsi disk) fails. (Note that I have tested this against libvirt-0.7.1 built from source as well--same results.) Here are some instructions to reproduce this issue: Create some small auxiliary storage to dynamically add to a running VM. (host) $ dd if=/dev/zero of=/tmp/foo bs=1M count=64 Boot a vm in under qemu-kvm-0.11.0 like this: (host) $ kvm -drive file=server.img,if=scsi,boot=on -boot c To reproduce the problem, you must use if=scsi. This problem manifests itself when subsequent scsi storage devices are added. If the device is the first scsi device on the system, then it would succeed. It's the 2nd scsi device that causes the problem. Once the system is booted, login and in the vm, load this module: (vm) $ sudo modprobe acpiphp Check that the acpiphp slots are loaded in dmesg. And note that there is only one /dev/sd[a-z] device. Now, drop to the qemu console with ctrl-alt-2, and add the storage: (qemu) pci_add auto storage file=/tmp/foo,if=scsi OK domain 0, bus 0, slot 6, function 0 Switch back to the vm linux shell with ctrl-alt-1, and look at the dmesg output. (vm) $ dmesg | tail -n 12 [ 44.033397] pci 0000:00:06.0: reg 10 io port: [0x00-0xff] [ 44.033443] pci 0000:00:06.0: reg 14 32bit mmio: [0x000000-0x0003ff] [ 44.033486] pci 0000:00:06.0: reg 18 32bit mmio: [0x000000-0x001fff] [ 44.033899] pci 0000:00:02.0: BAR 6: bogus alignment [0x0-0x0] flags 0x2 [ 44.033975] decode_hpp: Could not get hotplug parameters. Use defaults [ 44.042277] sym53c8xx 0000:00:06.0: enabling device (0000 -> 0003) [ 44.043230] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11 [ 44.043247] sym53c8xx 0000:00:06.0: PCI INT A -> Link[LNKB] -> GSI 11 (level, high) -> IRQ 11 [ 44.045237] sym1: <895a> rev 0x0 at pci 0000:00:06.0 irq 11 [ 44.047586] sym1: No NVRAM, ID 7, Fast-40, LVD, parity checking [ 44.055399] sym1: SCSI BUS has been reset. [ 44.063329] scsi3 : sym-2.2.3 More importantly, note that no /dev/sd[b-z] device shows up. If you now drop to the qemu console with ctrl-alt-2, and do this: (qemu) pci_add auto storage file=/tmp/foo,if=virtio OK domain 0, bus 0, slot 7, function 0 Going back to the vm with ctrl-alt-1, you can now see a new /dev/vda device registered. Now, all of that said, it is actually possible to hot-add a second scsi device. However, as far as I can tell, this method is not yet supported by libvirt. It looks to me that with modern qemu, you have to do it this way: Drop to a qemu console with ctrl-alt-2. Get the address of the current scsi bus: (qemu) info pci Look for "SCSI Controller". In my case, it's on Bus 0, device 4, function 0 Now instead of pci_add, use drive_add (qemu) drive_add 0:4 file=/tmp/foo,if=scsi OK bus 0, unit 1 This is not ideal, however, as I tried re-scan the scsi bus with rescan-scsi-bus.sh (from scsitools) without luck. It did not pick up the new sdb device. However, I did reboot the vm, and voila, I now have /dev/sdb. I would like some advice on how to proceed with this bug, and where the solution lies...in qemu or in libvirt. Ultimately, I would like the behavior we had in our previous release with kvm-84 and libvirt-0.6.1, where we could dynamically add scsi devices without a problem, using: pci_add 1 storage file=/tmp/foo,if=scsi Can anyone else reproduce this? Is this considered a regression by anyone else? Where should I look to solve this, in libvirt, or in qemu? -- :-Dustin

On Thu, Oct 08, 2009 at 10:57:55PM -0500, Dustin Kirkland wrote:
Now, all of that said, it is actually possible to hot-add a second scsi device. However, as far as I can tell, this method is not yet supported by libvirt. It looks to me that with modern qemu, you have to do it this way:
Drop to a qemu console with ctrl-alt-2. Get the address of the current scsi bus: (qemu) info pci Look for "SCSI Controller". In my case, it's on Bus 0, device 4, function 0
Now instead of pci_add, use drive_add (qemu) drive_add 0:4 file=/tmp/foo,if=scsi OK bus 0, unit 1
That is correct - the SCSI driver hotplug in libvirt is not implemented in the right way. If you specify multiple SCSI devices at boot, they all get on one controller, if you hotplug multiple SCSI devices, we're hotplugging a new SCSI controller per disk. This is clearly not good, because when you then reboot, all those controllers are merged back into one. There is a guy who is working on implementing the correct SCSI hotplug approach for libvirt, that is still work in progress though. The most recent patches were here: http://www.redhat.com/archives/libvir-list/2009-September/msg00551.html We will ultimately support hotplug of both drives, and drive controllers independantly, giving apps/users the flexibility they need.
I would like some advice on how to proceed with this bug, and where the solution lies...in qemu or in libvirt. Ultimately, I would like the behavior we had in our previous release with kvm-84 and libvirt-0.6.1, where we could dynamically add scsi devices without a problem, using: pci_add 1 storage file=/tmp/foo,if=scsi
Can anyone else reproduce this? Is this considered a regression by anyone else? Where should I look to solve this, in libvirt, or in qemu?
Independently of what I said about libvirt not implementing SCSI hotplug with the right apporoach, the pci-add stuff should definitely work, so if it doesn't then this is a regression that needs to be fixed Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

I would like some advice on how to proceed with this bug, and where the solution lies...in qemu or in libvirt. Ultimately, I would like the behavior we had in our previous release with kvm-84 and libvirt-0.6.1, where we could dynamically add scsi devices without a problem, using: pci_add 1 storage file=/tmp/foo,if=scsi
Can anyone else reproduce this? Is this considered a regression by anyone else? Where should I look to solve this, in libvirt, or in qemu?
Independently of what I said about libvirt not implementing SCSI hotplug with the right apporoach, the pci-add stuff should definitely work, so if it doesn't then this is a regression that needs to be fixed
There's definitely a bug somewhere. When you add a second PCI controller, the controller shows up in the guest but the disk attached to the controller isn't visible. Rescanning the scsi bus does not make the disk appear either. Regards, Anthony Liguori

On Fri, Oct 9, 2009 at 9:38 AM, Anthony Liguori <anthony@codemonkey.ws> wrote:
I would like some advice on how to proceed with this bug, and where the solution lies...in qemu or in libvirt. Ultimately, I would like the behavior we had in our previous release with kvm-84 and libvirt-0.6.1, where we could dynamically add scsi devices without a problem, using: pci_add 1 storage file=/tmp/foo,if=scsi
Can anyone else reproduce this? Is this considered a regression by anyone else? Where should I look to solve this, in libvirt, or in qemu?
Independently of what I said about libvirt not implementing SCSI hotplug with the right apporoach, the pci-add stuff should definitely work, so if it doesn't then this is a regression that needs to be fixed
There's definitely a bug somewhere. When you add a second PCI controller, the controller shows up in the guest but the disk attached to the controller isn't visible. Rescanning the scsi bus does not make the disk appear either.
Okay, thanks for confirming this, Anthony. I have added a task against upstream QEMU at: * https://bugs.edge.launchpad.net/ubuntu/+source/qemu-kvm/+bug/432154 :-Dustin

On Fri, Oct 9, 2009 at 3:18 AM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Oct 08, 2009 at 10:57:55PM -0500, Dustin Kirkland wrote:
Now, all of that said, it is actually possible to hot-add a second scsi device. However, as far as I can tell, this method is not yet supported by libvirt. It looks to me that with modern qemu, you have to do it this way:
Drop to a qemu console with ctrl-alt-2. Get the address of the current scsi bus: (qemu) info pci Look for "SCSI Controller". In my case, it's on Bus 0, device 4, function 0
Now instead of pci_add, use drive_add (qemu) drive_add 0:4 file=/tmp/foo,if=scsi OK bus 0, unit 1
That is correct - the SCSI driver hotplug in libvirt is not implemented in the right way. If you specify multiple SCSI devices at boot, they all get on one controller, if you hotplug multiple SCSI devices, we're hotplugging a new SCSI controller per disk. This is clearly not good, because when you then reboot, all those controllers are merged back into one.
Okay, in our previous usage of this, rebooting didn't matter very much.
There is a guy who is working on implementing the correct SCSI hotplug approach for libvirt, that is still work in progress though. The most recent patches were here:
http://www.redhat.com/archives/libvir-list/2009-September/msg00551.html
We will ultimately support hotplug of both drives, and drive controllers independantly, giving apps/users the flexibility they need.
Thanks. I'll track those patches with interest. :-Dustin
participants (3)
-
Anthony Liguori
-
Daniel P. Berrange
-
Dustin Kirkland