[libvirt-users] Oracle RAC in libvirt+KVM environment
We wan't to setup two Oracle instance and make RAC work on them. Both VM are setup based on libvirt + KVM, we use a lvm lun which formated in qcow2 format and set the shareable properties in the disk driver like this: <disk type='block' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source dev='/dev/81035c32-d2e4-4aaf-82fa-3e76ae587586/ca18f6a5-7c98-46ea-b562-9424e68a52f3'/> <target dev='hda' bus='ide'/> <shareable/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> But when we use it as shared data disk like we use iscsi/FC disk for shared data disk, the disk can't be recognized as a share disk, and RAC can't be setup in the environment. Any body can give me some guild on this? I am wondering shareable in the libvirt config really makes the disk shareable? How can I setup the RAC environment in this situable? I found that vmware and vbox can do this according to a blog post, I am wondering if kvm has this ability to make a disk really shareable. -- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
Anybody have idea on it? I tried to set the disk as raw format, and retried the setup process, but still can't get through. On Fri, Aug 2, 2013 at 1:58 PM, Timon Wang <timonwst@gmail.com> wrote:
We wan't to setup two Oracle instance and make RAC work on them. Both VM are setup based on libvirt + KVM, we use a lvm lun which formated in qcow2 format and set the shareable properties in the disk driver like this:
<disk type='block' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source dev='/dev/81035c32-d2e4-4aaf-82fa-3e76ae587586/ca18f6a5-7c98-46ea-b562-9424e68a52f3'/> <target dev='hda' bus='ide'/> <shareable/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk>
But when we use it as shared data disk like we use iscsi/FC disk for shared data disk, the disk can't be recognized as a share disk, and RAC can't be setup in the environment.
Any body can give me some guild on this?
I am wondering shareable in the libvirt config really makes the disk shareable?
How can I setup the RAC environment in this situable?
I found that vmware and vbox can do this according to a blog post, I am wondering if kvm has this ability to make a disk really shareable.
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
On 08/08/2013 03:54 AM, Timon Wang wrote:
Anybody have idea on it?
I tried to set the disk as raw format, and retried the setup process, but still can't get through.
Caveat: I know nothing in particular about Oracle RAC, but... Assuming that RAC uses something like SCSI reservations in order to share the disk, I would guess it doesn't like the disk being on the IDE bus.
On Fri, Aug 2, 2013 at 1:58 PM, Timon Wang <timonwst@gmail.com> wrote:
We wan't to setup two Oracle instance and make RAC work on them. Both VM are setup based on libvirt + KVM, we use a lvm lun which formated in qcow2 format and set the shareable properties in the disk driver like this:
<disk type='block' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source dev='/dev/81035c32-d2e4-4aaf-82fa-3e76ae587586/ca18f6a5-7c98-46ea-b562-9424e68a52f3'/> <target dev='hda' bus='ide'/> <shareable/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk>
But when we use it as shared data disk like we use iscsi/FC disk for shared data disk, the disk can't be recognized as a share disk, and RAC can't be setup in the environment.
Any body can give me some guild on this?
I am wondering shareable in the libvirt config really makes the disk shareable?
How can I setup the RAC environment in this situable?
I found that vmware and vbox can do this according to a blog post, I am wondering if kvm has this ability to make a disk really shareable.
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
I have tryied change the disk bus to SCSI, add a SCSI controller whose model is virtio-scsi, still can't setup the RAC instance. I tried to use windows 2008 Failover Cluster feature to setup a a Failover Cluster instead, and I can't find any cluster disk to share between two nodes. So when Failover Cluster is setup, I can't add any Cluster disk to the Failover Cluster. Have I missed some thing? On Thu, Aug 8, 2013 at 10:04 PM, Don Koch <dkoch@verizon.com> wrote:
On 08/08/2013 03:54 AM, Timon Wang wrote:
Anybody have idea on it?
I tried to set the disk as raw format, and retried the setup process, but still can't get through.
Caveat: I know nothing in particular about Oracle RAC, but...
Assuming that RAC uses something like SCSI reservations in order to share the disk, I would guess it doesn't like the disk being on the IDE bus.
On Fri, Aug 2, 2013 at 1:58 PM, Timon Wang <timonwst@gmail.com> wrote:
We wan't to setup two Oracle instance and make RAC work on them. Both VM are setup based on libvirt + KVM, we use a lvm lun which formated in qcow2 format and set the shareable properties in the disk driver like this:
<disk type='block' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source dev='/dev/81035c32-d2e4-4aaf-82fa-3e76ae587586/ca18f6a5-7c98-46ea-b562-9424e68a52f3'/> <target dev='hda' bus='ide'/> <shareable/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk>
But when we use it as shared data disk like we use iscsi/FC disk for shared data disk, the disk can't be recognized as a share disk, and RAC can't be setup in the environment.
Any body can give me some guild on this?
I am wondering shareable in the libvirt config really makes the disk shareable?
How can I setup the RAC environment in this situable?
I found that vmware and vbox can do this according to a blog post, I am wondering if kvm has this ability to make a disk really shareable.
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
On Sat, Aug 10, 2013 at 11:14:39AM +0800, Timon Wang wrote:
I have tryied change the disk bus to SCSI, add a SCSI controller whose model is virtio-scsi, still can't setup the RAC instance.
I tried to use windows 2008 Failover Cluster feature to setup a a Failover Cluster instead, and I can't find any cluster disk to share between two nodes. So when Failover Cluster is setup, I can't add any Cluster disk to the Failover Cluster.
Have I missed some thing?
I'm not sure what SCSI-level requirements RAC or Failover Cluster have. If anyone knows which features are needed it would be possible to confirm whether they are supported under KVM. I expect this can only work if you are passing through a shared LUN. Can you describe your configuration? Stefan
Yes, SCSI bus likes pass through a shared LUN to the vm, and I am using a shared LUN for 'share' purpose. I found a post that vmware use lsilogic bus for the shared disk, but my qemu/kvm version can't support lsilogic bus. I'm tring to update qemu/kvm version for lsilogic bus support. On 8/12/13, Stefan Hajnoczi <stefanha@gmail.com> wrote:
On Sat, Aug 10, 2013 at 11:14:39AM +0800, Timon Wang wrote:
I have tryied change the disk bus to SCSI, add a SCSI controller whose model is virtio-scsi, still can't setup the RAC instance.
I tried to use windows 2008 Failover Cluster feature to setup a a Failover Cluster instead, and I can't find any cluster disk to share between two nodes. So when Failover Cluster is setup, I can't add any Cluster disk to the Failover Cluster.
Have I missed some thing?
I'm not sure what SCSI-level requirements RAC or Failover Cluster have.
If anyone knows which features are needed it would be possible to confirm whether they are supported under KVM.
I expect this can only work if you are passing through a shared LUN. Can you describe your configuration?
Stefan
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
On Mon, Aug 12, 2013 at 06:17:51PM +0800, Timon Wang wrote:
Yes, SCSI bus likes pass through a shared LUN to the vm, and I am using a shared LUN for 'share' purpose.
I found a post that vmware use lsilogic bus for the shared disk, but my qemu/kvm version can't support lsilogic bus.
I'm tring to update qemu/kvm version for lsilogic bus support.
Use virtio-scsi. The emulated LSI SCSI controller has known bugs and is not actively developed - don't be surprised if you hit issues with it. The question is still what commands RAC or Failover Clustering use. If you find that the software refuses to run, it could be because additional work is required to make it work on KVM. Stefan
Thanks, i have tried virtio-scsi and LSI SCSI controller, by using a local file or a LUN as disk. Both raw and qcow2 disk format are be verified, but no way to make windows Failover Cluster and Oracle RAC work correctly. Actually, Oracle RAC and Windows Failover Cluster needs "share" storage like iscsi disk and fc disk to work correctly. In iscsi situation, we can use ip network for Failover Cluster "share" storage, but when we use fc as a high performance solution, we can't use fc "share" storage direct in VM(Not LUN passthrough). As I google the windows Failover Cluster requirements, I found that this issue may be that windows checks only iscsi and fc drivers, the type of the disk may be check by the driver type. In KVM situation, we can't emulate a iscsi/fc disk for VM to use, so we can't setup a environment like this. I'm wondering, if we want't to make sure the applications and service in the VM to always work, it's there another way to do this in KVM environment. On Tue, Aug 13, 2013 at 5:18 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
On Mon, Aug 12, 2013 at 06:17:51PM +0800, Timon Wang wrote:
Yes, SCSI bus likes pass through a shared LUN to the vm, and I am using a shared LUN for 'share' purpose.
I found a post that vmware use lsilogic bus for the shared disk, but my qemu/kvm version can't support lsilogic bus.
I'm tring to update qemu/kvm version for lsilogic bus support.
Use virtio-scsi. The emulated LSI SCSI controller has known bugs and is not actively developed - don't be surprised if you hit issues with it.
The question is still what commands RAC or Failover Clustering use. If you find that the software refuses to run, it could be because additional work is required to make it work on KVM.
Stefan
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
Hi all, I found a article about Hyper-V virtual Fiber Channel, I think this will make Failover Cluster work if KVM has the same feature. http://technet.microsoft.com/en-us/library/hh831413.aspx Hyper-V uses NPIV for virtual Fiber Channel, I have read some article about KVM NPIV, but how can I config it with libvirt? Any body can show me some example? On Wed, Aug 14, 2013 at 2:33 PM, Timon Wang <timonwst@gmail.com> wrote:
Thanks, i have tried virtio-scsi and LSI SCSI controller, by using a local file or a LUN as disk. Both raw and qcow2 disk format are be verified, but no way to make windows Failover Cluster and Oracle RAC work correctly.
Actually, Oracle RAC and Windows Failover Cluster needs "share" storage like iscsi disk and fc disk to work correctly.
In iscsi situation, we can use ip network for Failover Cluster "share" storage, but when we use fc as a high performance solution, we can't use fc "share" storage direct in VM(Not LUN passthrough).
As I google the windows Failover Cluster requirements, I found that this issue may be that windows checks only iscsi and fc drivers, the type of the disk may be check by the driver type. In KVM situation, we can't emulate a iscsi/fc disk for VM to use, so we can't setup a environment like this.
I'm wondering, if we want't to make sure the applications and service in the VM to always work, it's there another way to do this in KVM environment.
On Tue, Aug 13, 2013 at 5:18 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
On Mon, Aug 12, 2013 at 06:17:51PM +0800, Timon Wang wrote:
Yes, SCSI bus likes pass through a shared LUN to the vm, and I am using a shared LUN for 'share' purpose.
I found a post that vmware use lsilogic bus for the shared disk, but my qemu/kvm version can't support lsilogic bus.
I'm tring to update qemu/kvm version for lsilogic bus support.
Use virtio-scsi. The emulated LSI SCSI controller has known bugs and is not actively developed - don't be surprised if you hit issues with it.
The question is still what commands RAC or Failover Clustering use. If you find that the software refuses to run, it could be because additional work is required to make it work on KVM.
Stefan
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
On Wed, Aug 14, 2013 at 04:40:44PM +0800, Timon Wang wrote:
I found a article about Hyper-V virtual Fiber Channel, I think this will make Failover Cluster work if KVM has the same feature. http://technet.microsoft.com/en-us/library/hh831413.aspx
Hyper-V uses NPIV for virtual Fiber Channel, I have read some article about KVM NPIV, but how can I config it with libvirt? Any body can show me some example?
From my limited knowledge about this, NPIV itself won't make clustering
A web search turns up this: https://docs.fedoraproject.org/en-US/Fedora/18/html/Virtualization_Administr... You can use this if the host has a supported Fibre Channel HBA and your image is on a SAN LUN. possible. RAC or Failure Cluster probably still require specific SCSI commands in order to work (like persistent reservations) and that's what needs to be investigated in order to figure out a solution. Stefan
Thanks. I have read the link you provide, there is another link which tells me to pass a NPIV discovery lun as a disk, this is seen as a local direct access disk in windows. RAC and Failure Cluster both consider this pass through disk as local disk, not a share disk, and the setup process failed. Hyper-v provides a virtual Fiber Channel implementation, so I wondering if kvm has the same solution like it. On Wed, Aug 14, 2013 at 10:02 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
On Wed, Aug 14, 2013 at 04:40:44PM +0800, Timon Wang wrote:
I found a article about Hyper-V virtual Fiber Channel, I think this will make Failover Cluster work if KVM has the same feature. http://technet.microsoft.com/en-us/library/hh831413.aspx
Hyper-V uses NPIV for virtual Fiber Channel, I have read some article about KVM NPIV, but how can I config it with libvirt? Any body can show me some example?
A web search turns up this:
https://docs.fedoraproject.org/en-US/Fedora/18/html/Virtualization_Administr...
You can use this if the host has a supported Fibre Channel HBA and your image is on a SAN LUN.
From my limited knowledge about this, NPIV itself won't make clustering possible. RAC or Failure Cluster probably still require specific SCSI commands in order to work (like persistent reservations) and that's what needs to be investigated in order to figure out a solution.
Stefan
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
Il 15/08/2013 12:01, Timon Wang ha scritto:
Thanks.
I have read the link you provide, there is another link which tells me to pass a NPIV discovery lun as a disk, this is seen as a local direct access disk in windows. RAC and Failure Cluster both consider this pass through disk as local disk, not a share disk, and the setup process failed.
Hyper-v provides a virtual Fiber Channel implementation, so I wondering if kvm has the same solution like it.
Can you include the XML file you are using for the domain? Paolo
My domain xml is like this: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>2008-2</name> <uuid>6325d8a5-468d-42e9-b5cb-9a04f5f34e80</uuid> <memory unit='KiB'>524288</memory> <currentMemory unit='KiB'>524288</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-1.4'>hvm</type> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='floppy'> <driver name='qemu' type='raw' cache='none'/> <target dev='fda' bus='fdc'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/home/images/win2008_2_sys'/> <target dev='hda' bus='ide'/> <boot order='3'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/isos/windows2008_64r2.iso'/> <target dev='sdc' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/fedora/q_disk'/> <target dev='sda' bus='virtio'/> <shareable/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> <controller type='fdc' index='0'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:71:20:ae'/> <source bridge='br0'/> <target dev='vport2'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:12:a0:fd'/> <source network='default'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <graphics type='spice' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <sound model='ac97'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='32768' heads='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-rtc-td-hack'/> </qemu:commandline> </domain> On 8/19/13, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 15/08/2013 12:01, Timon Wang ha scritto:
Thanks.
I have read the link you provide, there is another link which tells me to pass a NPIV discovery lun as a disk, this is seen as a local direct access disk in windows. RAC and Failure Cluster both consider this pass through disk as local disk, not a share disk, and the setup process failed.
Hyper-v provides a virtual Fiber Channel implementation, so I wondering if kvm has the same solution like it.
Can you include the XML file you are using for the domain?
Paolo
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
Right now, I found that Windows Failover Cluster needs SCSI-3 Persistent Reservation, I don't know where if virtio-scsi supports this, according to http://www.ovirt.org/Features/Virtio-SCSI I found this: limited flexibility: virtio-blk does not support all possible storage scenarios. For example, it does not allow SCSI passthrough or persistent reservations. In principle, virtio-scsi provides anything that the underlying SCSI target (be it physical storage, iSCSI or the in-kernel target) supports. virtio-blk does not support persistent reservations, but virtio-scsi may surpport. Another web page which archive mail lists http://www.spinics.net/lists/target-devel/msg01813.html, which request virtio-scsi to implement SPC-3 (persistent reservation) feature, but I can't found any more information about this. On Tue, Aug 20, 2013 at 2:00 PM, Timon Wang <timonwst@gmail.com> wrote:
My domain xml is like this:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>2008-2</name> <uuid>6325d8a5-468d-42e9-b5cb-9a04f5f34e80</uuid> <memory unit='KiB'>524288</memory> <currentMemory unit='KiB'>524288</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-1.4'>hvm</type> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <disk type='file' device='floppy'> <driver name='qemu' type='raw' cache='none'/> <target dev='fda' bus='fdc'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/home/images/win2008_2_sys'/> <target dev='hda' bus='ide'/> <boot order='3'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/isos/windows2008_64r2.iso'/> <target dev='sdc' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/fedora/q_disk'/> <target dev='sda' bus='virtio'/> <shareable/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> <controller type='fdc' index='0'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:71:20:ae'/> <source bridge='br0'/> <target dev='vport2'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:12:a0:fd'/> <source network='default'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <graphics type='spice' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <sound model='ac97'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </sound> <video> <model type='qxl' ram='65536' vram='32768' heads='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-rtc-td-hack'/> </qemu:commandline> </domain>
On 8/19/13, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 15/08/2013 12:01, Timon Wang ha scritto:
Thanks.
I have read the link you provide, there is another link which tells me to pass a NPIV discovery lun as a disk, this is seen as a local direct access disk in windows. RAC and Failure Cluster both consider this pass through disk as local disk, not a share disk, and the setup process failed.
Hyper-v provides a virtual Fiber Channel implementation, so I wondering if kvm has the same solution like it.
Can you include the XML file you are using for the domain?
Paolo
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
Il 20/08/2013 08:00, Timon Wang ha scritto:
<disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/home/images/win2008_2_sys'/> <target dev='hda' bus='ide'/> <boot order='3'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/isos/windows2008_64r2.iso'/> <target dev='sdc' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='block' device='disk'>
I'm not sure this will be enough, but if you want passthrough to the host device you should use device='lun' here. However, you still would not be able to issue SCSI reservations unless you run QEMU with the CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>"). Most important, it still would be unsafe to do this if the same device is passed to multiple virtual machines on the same host. You need to have NPIV and create separate virtual HBAs. Then each virtual machine should get a separate virtual HBA. Otherwise, persistent reservations are not attached to a particular virtual machine, but generically to the host.
<driver name='qemu' type='raw'/> <source dev='/dev/fedora/q_disk'/> <target dev='sda' bus='virtio'/>
You are not exposing a virtio-scsi disk here. You are exposing a virtio-blk disk. You can see this from the type='pci' address that libvirt gave to the disk. If you use bus='scsi', you will see that libvirt will use type='drive' for the address.
<controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller>
This is okay.
<qemu:commandline> <qemu:arg value='-rtc-td-hack'/> </qemu:commandline>
FWIW, this can be replaced with <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> </clock> (you already have the <clock> element, but no <timer> element inside). Paolo
</domain>
On 8/19/13, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 15/08/2013 12:01, Timon Wang ha scritto:
Thanks.
I have read the link you provide, there is another link which tells me to pass a NPIV discovery lun as a disk, this is seen as a local direct access disk in windows. RAC and Failure Cluster both consider this pass through disk as local disk, not a share disk, and the setup process failed.
Hyper-v provides a virtual Fiber Channel implementation, so I wondering if kvm has the same solution like it.
Can you include the XML file you are using for the domain?
Paolo
On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 08:00, Timon Wang ha scritto:
<disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/home/images/win2008_2_sys'/> <target dev='hda' bus='ide'/> <boot order='3'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/isos/windows2008_64r2.iso'/> <target dev='sdc' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='block' device='disk'>
I'm not sure this will be enough, but if you want passthrough to the host device you should use device='lun' here. However, you still would not be able to issue SCSI reservations unless you run QEMU with the CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
After change the libvirt xml like this: <disk type='block' device='lun' rawio='yes'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/> <target dev='sda' bus='scsi'/> <shareable/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> I got these errors: char device redirected to /dev/pts/1 (label charserial0) qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: scsi-block: INQUIRY failed qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: Device 'scsi-block' could not be initialized
Most important, it still would be unsafe to do this if the same device is passed to multiple virtual machines on the same host. You need to have NPIV and create separate virtual HBAs. Then each virtual machine should get a separate virtual HBA. Otherwise, persistent reservations are not attached to a particular virtual machine, but generically to the host.
How to use NPIV virtual HBAs with libvirt xml configurations? I can define nodedev, but have no idea about how to pass the nodedev to the vm.
<driver name='qemu' type='raw'/> <source dev='/dev/fedora/q_disk'/> <target dev='sda' bus='virtio'/>
You are not exposing a virtio-scsi disk here. You are exposing a virtio-blk disk. You can see this from the type='pci' address that libvirt gave to the disk.
If you use bus='scsi', you will see that libvirt will use type='drive' for the address.
<controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller>
This is okay.
<qemu:commandline> <qemu:arg value='-rtc-td-hack'/> </qemu:commandline>
FWIW, this can be replaced with
<clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> </clock>
(you already have the <clock> element, but no <timer> element inside).
Thanks for this tip.
Paolo
</domain>
On 8/19/13, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 15/08/2013 12:01, Timon Wang ha scritto:
Thanks.
I have read the link you provide, there is another link which tells me to pass a NPIV discovery lun as a disk, this is seen as a local direct access disk in windows. RAC and Failure Cluster both consider this pass through disk as local disk, not a share disk, and the setup process failed.
Hyper-v provides a virtual Fiber Channel implementation, so I wondering if kvm has the same solution like it.
Can you include the XML file you are using for the domain?
Paolo
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
Il 20/08/2013 11:59, Timon Wang ha scritto:
On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 08:00, Timon Wang ha scritto:
<disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/home/images/win2008_2_sys'/> <target dev='hda' bus='ide'/> <boot order='3'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/isos/windows2008_64r2.iso'/> <target dev='sdc' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='block' device='disk'>
I'm not sure this will be enough, but if you want passthrough to the host device you should use device='lun' here. However, you still would not be able to issue SCSI reservations unless you run QEMU with the CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
After change the libvirt xml like this: <disk type='block' device='lun' rawio='yes'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/> <target dev='sda' bus='scsi'/> <shareable/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> I got these errors: char device redirected to /dev/pts/1 (label charserial0) qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: scsi-block: INQUIRY failed qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: Device 'scsi-block' could not be initialized
Can you do # ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk # sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk ? Paolo
[root@localhost /]# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk lrwxrwxrwx. 1 root root 8 8月 20 17:38 /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk -> ../dm-13 [root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300 This lun is from a vg build based on iscsi target. [root@localhost /]# libvirtd --version libvirtd (libvirt) 1.0.5 [root@localhost /]# qemu-kvm --version QEMU emulator version 1.4.1, Copyright (c) 2003-2008 Fabrice Bellard [root@localhost /]# uname -a Linux localhost.localdomain 3.9.2-301.fc19.x86_64 #1 SMP Mon May 13 12:36:24 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux On Tue, Aug 20, 2013 at 6:16 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 11:59, Timon Wang ha scritto:
On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 08:00, Timon Wang ha scritto:
<disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/home/images/win2008_2_sys'/> <target dev='hda' bus='ide'/> <boot order='3'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/isos/windows2008_64r2.iso'/> <target dev='sdc' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='block' device='disk'>
I'm not sure this will be enough, but if you want passthrough to the host device you should use device='lun' here. However, you still would not be able to issue SCSI reservations unless you run QEMU with the CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
After change the libvirt xml like this: <disk type='block' device='lun' rawio='yes'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/> <target dev='sda' bus='scsi'/> <shareable/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> I got these errors: char device redirected to /dev/pts/1 (label charserial0) qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: scsi-block: INQUIRY failed qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: Device 'scsi-block' could not be initialized
Can you do
# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk # sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
?
Paolo
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
Il 20/08/2013 12:42, Timon Wang ha scritto:
[root@localhost /]# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk lrwxrwxrwx. 1 root root 8 8月 20 17:38 /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk -> ../dm-13 [root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300
This lun is from a vg build based on iscsi target.
If it is a logical volume, you cannot pass it as a LUN to the guest. Only the whole iSCSI LUN can be passed as a LUN. Paolo
[root@localhost /]# libvirtd --version libvirtd (libvirt) 1.0.5 [root@localhost /]# qemu-kvm --version QEMU emulator version 1.4.1, Copyright (c) 2003-2008 Fabrice Bellard [root@localhost /]# uname -a Linux localhost.localdomain 3.9.2-301.fc19.x86_64 #1 SMP Mon May 13 12:36:24 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
On Tue, Aug 20, 2013 at 6:16 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 11:59, Timon Wang ha scritto:
On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 08:00, Timon Wang ha scritto:
<disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/home/images/win2008_2_sys'/> <target dev='hda' bus='ide'/> <boot order='3'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/isos/windows2008_64r2.iso'/> <target dev='sdc' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='block' device='disk'>
I'm not sure this will be enough, but if you want passthrough to the host device you should use device='lun' here. However, you still would not be able to issue SCSI reservations unless you run QEMU with the CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
After change the libvirt xml like this: <disk type='block' device='lun' rawio='yes'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/> <target dev='sda' bus='scsi'/> <shareable/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> I got these errors: char device redirected to /dev/pts/1 (label charserial0) qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: scsi-block: INQUIRY failed qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: Device 'scsi-block' could not be initialized
Can you do
# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk # sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
?
Paolo
Thanks, the whole iSCSI LUN have been passed to the VM. But I test it with scsicmd, and found that the driver may be not support SPC-3, but if i use this by microsoft iscsi initiator, I can pass all the scsi3_test tests. Tool can be found here: http://www.symantec.com/business/support/index?page=content&id=TECH72086 It's this means that the scsi passthrough windows driver does not support SPC-3 feature, I have read a post about this, it says if support this we should change both the implementation and the documents in virtio spec. I am new to this list, so I don't know what is the situation right now? Would somebody please give me some advise on it? On Tue, Aug 20, 2013 at 6:49 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 12:42, Timon Wang ha scritto:
[root@localhost /]# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk lrwxrwxrwx. 1 root root 8 8月 20 17:38 /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk -> ../dm-13 [root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300
This lun is from a vg build based on iscsi target.
If it is a logical volume, you cannot pass it as a LUN to the guest. Only the whole iSCSI LUN can be passed as a LUN.
Paolo
[root@localhost /]# libvirtd --version libvirtd (libvirt) 1.0.5 [root@localhost /]# qemu-kvm --version QEMU emulator version 1.4.1, Copyright (c) 2003-2008 Fabrice Bellard [root@localhost /]# uname -a Linux localhost.localdomain 3.9.2-301.fc19.x86_64 #1 SMP Mon May 13 12:36:24 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
On Tue, Aug 20, 2013 at 6:16 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 11:59, Timon Wang ha scritto:
On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 08:00, Timon Wang ha scritto:
<disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/home/images/win2008_2_sys'/> <target dev='hda' bus='ide'/> <boot order='3'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/home/isos/windows2008_64r2.iso'/> <target dev='sdc' bus='ide'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='block' device='disk'>
I'm not sure this will be enough, but if you want passthrough to the host device you should use device='lun' here. However, you still would not be able to issue SCSI reservations unless you run QEMU with the CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
After change the libvirt xml like this: <disk type='block' device='lun' rawio='yes'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/> <target dev='sda' bus='scsi'/> <shareable/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> I got these errors: char device redirected to /dev/pts/1 (label charserial0) qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: scsi-block: INQUIRY failed qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: Device 'scsi-block' could not be initialized
Can you do
# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk # sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
?
Paolo
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
I found when I use "scsicmd -d1 -s13" test command to test the "controller bus reset" request, there will be a blue screen on windows 2008 r2. The error code is : BugCheck D1, {4, a, 0, fffff8800154dd06} 1: kd> !analyze -v ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1) An attempt was made to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that is too high. This is usually caused by drivers using improper addresses. If kernel debugger is available get stack backtrace. Arguments: Arg1: 0000000000000004, memory referenced Arg2: 000000000000000a, IRQL Arg3: 0000000000000000, value 0 = read operation, 1 = write operation Arg4: fffff8800154dd06, address which referenced memory Debugging Details: ------------------ Page 17c41 not present in the dump file. Type ".hh dbgerr004" for details READ_ADDRESS: 0000000000000004 CURRENT_IRQL: a FAULTING_IP: vioscsi+1d06 fffff880`0154dd06 458b4804 mov r9d,dword ptr [r8+4] DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT BUGCHECK_STR: 0xD1 PROCESS_NAME: scsicmd.exe TRAP_FRAME: fffff880009f7670 -- (.trap 0xfffff880009f7670) NOTE: The trap frame does not contain all registers. Some register values may be zeroed or incorrect. rax=0000000000000002 rbx=0000000000000000 rcx=fffffa800065e738 rdx=fffffa800065e8f8 rsi=0000000000000000 rdi=0000000000000000 rip=fffff8800154dd06 rsp=fffff880009f7800 rbp=fffffa800065e8f8 r8=0000000000000000 r9=0000000000000000 r10=fffffa80009155b0 r11=fffff880009f7848 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl zr na po nc vioscsi+0x1d06: fffff880`0154dd06 458b4804 mov r9d,dword ptr [r8+4] ds:e630:0004=???????? Resetting default scope LAST_CONTROL_TRANSFER: from fffff800016ca469 to fffff800016caf00 STACK_TEXT: fffff880`009f7528 fffff800`016ca469 : 00000000`0000000a 00000000`00000004 00000000`0000000a 00000000`00000000 : nt!KeBugCheckEx fffff880`009f7530 fffff800`016c90e0 : 00000000`00000000 fffffa80`009151b0 fffffa80`0155a290 fffff880`01339110 : nt!KiBugCheckDispatch+0x69 fffff880`009f7670 fffff880`0154dd06 : 00000000`00000001 fffff880`0154dcec fffffa80`009151b0 fffff880`01323934 : nt!KiPageFault+0x260 fffff880`009f7800 fffff880`0132abcf : fffffa80`009151b0 fffffa80`0065e8f8 fffffa80`0065e738 00000000`00000001 : vioscsi+0x1d06 fffff880`009f7850 fffff880`0154d971 : 00000000`00000001 00000000`00000001 00000000`002d5000 fffffa80`00925000 : storport!StorPortSynchronizeAccess+0x4f fffff880`009f7890 fffff880`01323a0c : fffffa80`00000fb1 fffffa80`0155a200 00000000`002d5000 fffffa80`01576010 : vioscsi+0x1971 fffff880`009f78d0 fffff880`01333adf : fffffa80`006eeb30 fffffa80`006e2070 00000000`00000000 00000000`00000801 : storport!RaCallMiniportResetBus+0x1c fffff880`009f7900 fffff880`01333b68 : fffffa80`0155a290 fffffa80`006b39f0 00000040`00000000 00000000`00000000 : storport!RaidAdapterResetBus+0x2f fffff880`009f7950 fffff880`0136de0b : 00000000`20206f49 00000000`00000001 00000000`00000001 00000000`20206f49 : storport!RaidAdapterStorageResetBusIoctl+0x28 fffff880`009f7980 fffff880`0136d1d0 : fffff880`01339110 fffffa80`00915060 00000000`00000000 fffffa80`006e2070 : storport! ?? ::NNGAKEGL::`string'+0x3c8 fffff880`009f79d0 fffff800`019e33a7 : fffffa80`006e2070 fffff880`009f7ca0 fffffa80`006e2070 fffffa80`0155a290 : storport!RaDriverDeviceControlIrp+0x90 fffff880`009f7a10 fffff800`019e3c06 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0x607 fffff880`009f7b40 fffff800`016ca153 : 00000000`001aeb01 00000000`00000001 00000000`001aeba0 fffff800`019db152 : nt!NtDeviceIoControlFile+0x56 fffff880`009f7bb0 00000000`77a2ff2a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 00000000`001af1d8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x77a2ff2a STACK_COMMAND: kb FOLLOWUP_IP: vioscsi+1d06 fffff880`0154dd06 458b4804 mov r9d,dword ptr [r8+4] SYMBOL_STACK_INDEX: 3 SYMBOL_NAME: vioscsi+1d06 FOLLOWUP_NAME: MachineOwner MODULE_NAME: vioscsi IMAGE_NAME: vioscsi.sys DEBUG_FLR_IMAGE_TIMESTAMP: 5200724f FAILURE_BUCKET_ID: X64_0xD1_vioscsi+1d06 BUCKET_ID: X64_0xD1_vioscsi+1d06 Followup: MachineOwner --------- On Tue, Aug 20, 2013 at 7:43 PM, Timon Wang <timonwst@gmail.com> wrote:
Thanks, the whole iSCSI LUN have been passed to the VM.
But I test it with scsicmd, and found that the driver may be not support SPC-3, but if i use this by microsoft iscsi initiator, I can pass all the scsi3_test tests.
Tool can be found here: http://www.symantec.com/business/support/index?page=content&id=TECH72086
It's this means that the scsi passthrough windows driver does not support SPC-3 feature, I have read a post about this, it says if support this we should change both the implementation and the documents in virtio spec.
I am new to this list, so I don't know what is the situation right now?
Would somebody please give me some advise on it?
On Tue, Aug 20, 2013 at 6:49 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 12:42, Timon Wang ha scritto:
[root@localhost /]# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk lrwxrwxrwx. 1 root root 8 8月 20 17:38 /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk -> ../dm-13 [root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300
This lun is from a vg build based on iscsi target.
If it is a logical volume, you cannot pass it as a LUN to the guest. Only the whole iSCSI LUN can be passed as a LUN.
Paolo
[root@localhost /]# libvirtd --version libvirtd (libvirt) 1.0.5 [root@localhost /]# qemu-kvm --version QEMU emulator version 1.4.1, Copyright (c) 2003-2008 Fabrice Bellard [root@localhost /]# uname -a Linux localhost.localdomain 3.9.2-301.fc19.x86_64 #1 SMP Mon May 13 12:36:24 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
On Tue, Aug 20, 2013 at 6:16 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 11:59, Timon Wang ha scritto:
On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 08:00, Timon Wang ha scritto: > <disk type='file' device='disk'> > <driver name='qemu' type='raw' cache='none'/> > <source file='/home/images/win2008_2_sys'/> > <target dev='hda' bus='ide'/> > <boot order='3'/> > <address type='drive' controller='0' bus='0' target='0' unit='0'/> > </disk> > <disk type='file' device='cdrom'> > <driver name='qemu' type='raw'/> > <source file='/home/isos/windows2008_64r2.iso'/> > <target dev='sdc' bus='ide'/> > <readonly/> > <boot order='1'/> > <address type='drive' controller='0' bus='1' target='0' unit='0'/> > </disk> > <disk type='block' device='disk'>
I'm not sure this will be enough, but if you want passthrough to the host device you should use device='lun' here. However, you still would not be able to issue SCSI reservations unless you run QEMU with the CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
After change the libvirt xml like this: <disk type='block' device='lun' rawio='yes'> <driver name='qemu' type='raw' cache='none'/> <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/> <target dev='sda' bus='scsi'/> <shareable/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> I got these errors: char device redirected to /dev/pts/1 (label charserial0) qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: scsi-block: INQUIRY failed qemu-system-x86_64: -device scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0: Device 'scsi-block' could not be initialized
Can you do
# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk # sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
?
Paolo
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
Il 20/08/2013 13:43, Timon Wang ha scritto:
Thanks, the whole iSCSI LUN have been passed to the VM.
But I test it with scsicmd, and found that the driver may be not support SPC-3, but if i use this by microsoft iscsi initiator, I can pass all the scsi3_test tests.
If you are passing the LUN to the VM with device='lun', the driver and VMM do not interpret any SCSI command. You should see exactly the same data as in the host, which includes support for SPC-3:
[root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300
Can you try using a Linux VM and executing sg_inq in the VM? Paolo
From the fedora 19 host: [root@fedora ~]# sg_inq /dev/sdc standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
From the fedora 19 vm: [root@fedoravm ~]# sg_inq /dev/sdb standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
The result from fedora 19 host and fedora 19 vm are the same. It's that means I got a wrong windows vm scsi pass-through driver? Or is there any tool like sg_inq in windows 2008? On Tue, Aug 20, 2013 at 8:09 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 20/08/2013 13:43, Timon Wang ha scritto:
Thanks, the whole iSCSI LUN have been passed to the VM.
But I test it with scsicmd, and found that the driver may be not support SPC-3, but if i use this by microsoft iscsi initiator, I can pass all the scsi3_test tests.
If you are passing the LUN to the VM with device='lun', the driver and VMM do not interpret any SCSI command. You should see exactly the same data as in the host, which includes support for SPC-3:
[root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300
Can you try using a Linux VM and executing sg_inq in the VM?
Paolo
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
Il 21/08/2013 04:11, Timon Wang ha scritto:
From the fedora 19 host: [root@fedora ~]# sg_inq /dev/sdc standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
From the fedora 19 vm: [root@fedoravm ~]# sg_inq /dev/sdb standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
The result from fedora 19 host and fedora 19 vm are the same. It's that means I got a wrong windows vm scsi pass-through driver? Or is there any tool like sg_inq in windows 2008?
Yeah, there's something weird in the Windows VM. sg_inq should be available for Windows too, but I don't know where to get a precompiled binary from. Paolo
On Wed, 2013-08-21 at 11:09 +0200, Paolo Bonzini wrote:
Il 21/08/2013 04:11, Timon Wang ha scritto:
From the fedora 19 host: [root@fedora ~]# sg_inq /dev/sdc standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
From the fedora 19 vm: [root@fedoravm ~]# sg_inq /dev/sdb standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
The result from fedora 19 host and fedora 19 vm are the same. It's that means I got a wrong windows vm scsi pass-through driver? Or is there any tool like sg_inq in windows 2008?
Yeah, there's something weird in the Windows VM.
sg_inq should be available for Windows too, but I don't know where to get a precompiled binary from.
AFAIK, the latest sg3-utils build for MSFT is here: http://sg.danny.cz/sg/p/sg3_utils-1.36exe.zip --nab
Thanks Nicholas. I found that scsicmd can't pass all the scsi3_test but the result of sg_inq is the same as it in the host. I am absolutely confused about this situation. Am I missed some information about it? On Wed, Aug 21, 2013 at 5:40 PM, Nicholas A. Bellinger <nab@linux-iscsi.org> wrote:
On Wed, 2013-08-21 at 11:09 +0200, Paolo Bonzini wrote:
Il 21/08/2013 04:11, Timon Wang ha scritto:
From the fedora 19 host: [root@fedora ~]# sg_inq /dev/sdc standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
From the fedora 19 vm: [root@fedoravm ~]# sg_inq /dev/sdb standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x05 [SPC-3] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=0 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 [BQue=0] EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=1 Sync=1 Linked=0 [TranDis=0] CmdQue=1 length=36 (0x24) Peripheral device type: disk Vendor identification: MacroSAN Product identification: LU Product revision level: 1.0 Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
The result from fedora 19 host and fedora 19 vm are the same. It's that means I got a wrong windows vm scsi pass-through driver? Or is there any tool like sg_inq in windows 2008?
Yeah, there's something weird in the Windows VM.
sg_inq should be available for Windows too, but I don't know where to get a precompiled binary from.
AFAIK, the latest sg3-utils build for MSFT is here:
http://sg.danny.cz/sg/p/sg3_utils-1.36exe.zip
--nab
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
Il 22/08/2013 09:46, Timon Wang ha scritto:
Thanks Nicholas.
I found that scsicmd can't pass all the scsi3_test but the result of sg_inq is the same as it in the host.
I am absolutely confused about this situation. Am I missed some information about it?
I am also confused. You need to understand the limitations that the clustering software is putting. Can you also try using the megaraid sas controller ("lsisas1078" in libvirt) instead of virtio-scsi? Paolo
I don't know if lsisas1078 controller in qemu is not work property or I did not configure it correct. Sometimes I can see the lun which passed to the vm, sometimes not. I have tried that controller before, and sometimes the vm will crash because qemu process quit, but I did not get the detail logs about the process quit. I am wondering if lsisas1078 is still not stable or I did not config it right. On Thu, Aug 22, 2013 at 5:02 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 22/08/2013 09:46, Timon Wang ha scritto:
Thanks Nicholas.
I found that scsicmd can't pass all the scsi3_test but the result of sg_inq is the same as it in the host.
I am absolutely confused about this situation. Am I missed some information about it?
I am also confused. You need to understand the limitations that the clustering software is putting.
Can you also try using the megaraid sas controller ("lsisas1078" in libvirt) instead of virtio-scsi?
Paolo
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
I found that scsicmd can't pass all the scsi3_test but the result of sg_inq is the same as it in the host.
I am absolutely confused about this situation. Am I missed some information about it?
I guess this is caused by the lack of capability. Please check if enough capability was added to kvm process by the following steps. 1. Check the pid of kvm process. # ps -C qemu-system-x86_64 -o pid= 5177 2. Check the capability for the process. # getpcaps 5177 Capabilities for `5177': = cap_sys_rawio+i In my fedora19 environment, as seen in above, only cap_sys_rawio+i was added with rawio='yes'. Even though, cap_sys_rawio+ep is required to pass-through SCSI Reservation from the guest. Note that I succeeded to pass-through SCSI Reservation with the following steps in my environment, not a Windows guest though. 1. Stop the guest. 2. Add CAP_SYS_RAWIO(effective, permitted) to qemu-kvm. # setcap cap_sys_rawio=ep /usr/bin/qemu-system-x86_64 3. Start the guest. However, I don't think this is the right way to workaround it, because it gives cap_sys_rawio+ep to all the kvm processes executed from this binary. I believe following patches, which are not merged yet, are trying to solve this problem in a different approach. - [PATCH v3 part2] Add per-device sysfs knob to enable unrestricted, unprivileged SG_IO https://lkml.org/lkml/2013/5/23/294 - [RFC PATCH 0/4] SG_IO filtering via sysfs and minimal whitelist https://lkml.org/lkml/2013/5/27/230 Any comments on this? Masaki
Il 27/08/2013 12:29, Masaki Kimura ha scritto:
I believe following patches, which are not merged yet, are trying to solve this problem in a different approach. - [PATCH v3 part2] Add per-device sysfs knob to enable unrestricted, unprivileged SG_IO https://lkml.org/lkml/2013/5/23/294 - [RFC PATCH 0/4] SG_IO filtering via sysfs and minimal whitelist https://lkml.org/lkml/2013/5/27/230
Any comments on this?
Yes, that's correct. Paolo
Thanks, all the test in SCSICMD.exe can pass now, when I set the ep caps on qemu process. We found that sg_persist --read-keys can't get the same result in VM and host, and now we get the same result in both VM and host mode. But Windows 2008 Failover Cluster Still can't get the cluster disk configured, with some error said that the driver can't support the cluster. Thanks, I think I have dig into the windows Failover Cluster to find why it can't support cluster, I will ask somebody in MSDN. On Tue, Aug 27, 2013 at 7:07 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
Il 27/08/2013 12:29, Masaki Kimura ha scritto:
I believe following patches, which are not merged yet, are trying to solve this problem in a different approach. - [PATCH v3 part2] Add per-device sysfs knob to enable unrestricted, unprivileged SG_IO https://lkml.org/lkml/2013/5/23/294 - [RFC PATCH 0/4] SG_IO filtering via sysfs and minimal whitelist https://lkml.org/lkml/2013/5/27/230
Any comments on this?
Yes, that's correct.
Paolo
-- Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW Blog: http://www.nohouse.net
On Fri, Aug 02, 2013 at 01:58:24PM +0800, Timon Wang wrote:
We wan't to setup two Oracle instance and make RAC work on them. Both VM are setup based on libvirt + KVM, we use a lvm lun which formated in qcow2 format and set the shareable properties in the disk driver like this:
<disk type='block' device='disk'> <driver name='qemu' type='qcow2' cache='none'/>
qcow2 is not cluster-aware, it cannot be opened by multiple VMs at the same time. You must use raw. Stefan
participants (6)
-
Don Koch -
Masaki Kimura -
Nicholas A. Bellinger -
Paolo Bonzini -
Stefan Hajnoczi -
Timon Wang