[libvirt-users] Problem booting guest with more than 8 disks

Hello All, On my host machine, I'm using kvm, libvirt, ceph and ubuntu versions as follows: 2003-2008 Fabrice Bellard x86_64 x86_64 x86_64 GNU/Linux I'm encountering an issue when attempting to boot a VM with more than eight disks, the boot disk is not found and the vm fails to boot with an error "No bootable device". This happens with Archlinux, Ubuntu. and Fedora images. I'm working on a tool [1] to build virtual machines, it creates rbd images named $hostname-os and $hostname-storageX (where X is the "additional disk number" starting at 0, a vm with seven disks would have one named -os and six named -storage{0..5})--the os disk is a clone of a protected COW RBD snapshot--the the tool defines the xml and uses Sys::Virt to define and boot the virtual machine. All os types use grub, ubuntu and fedora use an mbr and arch uses gpt, I'm not certain that matters as vms with less than six disks boot no problem. I've included both my generated and dumped xml for both my booting and non booting vms. The only consistent difference between the vm deployments is the addition on $hostname-storage6. Is there perhaps a way to specify which hd is the boot device? I appreciate any assistance. Thanks, Jon A [1] https://github.com/three18ti/Build-VM non Booting VM: ######################################################### # Generated: | <domain type='kvm'> | <name>arch_test2</name> | <memory unit='KiB'>4194304</memory> | <currentMemory unit='KiB'>4194304</currentMemory> | <vcpu placement='static'>2</vcpu> | <os> | <type arch='x86_64' machine='pc-i440fx-1.4'>hvm</type> | <boot dev='hd'/> | <bootmenu enable='no'/> | </os> | <features> | <acpi/> | <apic/> | <pae/> | </features> | <clock offset='utc'/> | <on_poweroff>destroy</on_poweroff> | <on_reboot>restart</on_reboot> | <on_crash>restart</on_crash> | <devices> | <emulator>/usr/bin/kvm</emulator> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-os'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vda' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage0'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdb' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage1'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdc' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage2'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdd' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage3'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vde' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage4'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdf' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage5'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdg' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage6'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdh' bus='virtio' /> | </disk> | <interface type='bridge'> | <source bridge='ovsbr0'/> | <virtualport type='openvswitch'> | </virtualport> | <model type='virtio'/> | </interface> | | <controller type='usb' index='0'> | </controller> | <controller type='ide' index='0'> | </controller> | <controller type='virtio-serial' index='0'> | </controller> | <serial type='pty'> | <target port='0'/> | </serial> | <console type='pty'> | <target type='serial' port='0'/> | </console> | <input type='mouse' bus='ps2'/> | <graphics type='vnc' port='-1' autoport='yes'/> | <sound model='ich6'> | </sound> | | <video> | <model type='cirrus' vram='9216' heads='1'/> | </video> | | <memballoon model='virtio'> | </memballoon> | | </devices> | </domain> ##### ------------------------------------------------------------------------------------------------------------- ##### # Dumped | <domain type='kvm' id='171'> | <name>arch_test2</name> | <uuid>fcff77f4-0a93-43f9-bf3a-e5863b787400</uuid> | <memory unit='KiB'>4194304</memory> | <currentMemory unit='KiB'>4194304</currentMemory> | <vcpu placement='static'>2</vcpu> | <resource> | <partition>/machine</partition> | </resource> | <os> | <type arch='x86_64' machine='pc-i440fx-1.4'>hvm</type> | <boot dev='hd'/> | <bootmenu enable='no'/> | </os> | <features> | <acpi/> | <apic/> | <pae/> | </features> | <clock offset='utc'/> | <on_poweroff>destroy</on_poweroff> | <on_reboot>restart</on_reboot> | <on_crash>restart</on_crash> | <devices> | <emulator>/usr/bin/kvm</emulator> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-os'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vda' bus='virtio'/> | <alias name='virtio-disk0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage0'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdb' bus='virtio'/> | <alias name='virtio-disk1'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage1'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdc' bus='virtio'/> | <alias name='virtio-disk2'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage2'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdd' bus='virtio'/> | <alias name='virtio-disk3'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage3'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vde' bus='virtio'/> | <alias name='virtio-disk4'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage4'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdf' bus='virtio'/> | <alias name='virtio-disk5'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage5'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdg' bus='virtio'/> | <alias name='virtio-disk6'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test2-storage6'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdh' bus='virtio'/> | <alias name='virtio-disk7'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> | </disk> | <controller type='usb' index='0'> | <alias name='usb0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> | </controller> | <controller type='ide' index='0'> | <alias name='ide0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> | </controller> | <controller type='virtio-serial' index='0'> | <alias name='virtio-serial0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> | </controller> | <controller type='pci' index='0' model='pci-root'> | <alias name='pci0'/> | </controller> | <interface type='bridge'> | <mac address='52:54:00:c2:64:72'/> | <source bridge='ovsbr0'/> | <virtualport type='openvswitch'> | <parameters interfaceid='51324f0c-e98f-419e-aa82-ef9942c27eea'/> | </virtualport> | <target dev='vnet4'/> | <model type='virtio'/> | <alias name='net0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> | </interface> | <serial type='pty'> | <source path='/dev/pts/13'/> | <target port='0'/> | <alias name='serial0'/> | </serial> | <console type='pty' tty='/dev/pts/13'> | <source path='/dev/pts/13'/> | <target type='serial' port='0'/> | <alias name='serial0'/> | </console> | <input type='mouse' bus='ps2'/> | <graphics type='vnc' port='5904' autoport='yes' listen='127.0.0.1'> | <listen type='address' address='127.0.0.1'/> | </graphics> | <sound model='ich6'> | <alias name='sound0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> | </sound> | <video> | <model type='cirrus' vram='9216' heads='1'/> | <alias name='video0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> | </video> | <memballoon model='virtio'> | <alias name='balloon0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/> | </memballoon> | </devices> | <seclabel type='dynamic' model='apparmor' relabel='yes'> | <label>libvirt-fcff77f4-0a93-43f9-bf3a-e5863b787400</label> | <imagelabel>libvirt-fcff77f4-0a93-43f9-bf3a-e5863b787400</imagelabel> | </seclabel> | </domain> -------------------------------------------------------------------------------------------------------------- ############################################################### ############################################################### # Booting VM: # Generated: | <domain type='kvm'> | <name>arch_test</name> | <memory unit='KiB'>4194304</memory> | <currentMemory unit='KiB'>4194304</currentMemory> | <vcpu placement='static'>2</vcpu> | <os> | <type arch='x86_64' machine='pc-i440fx-1.4'>hvm</type> | <boot dev='hd'/> | <bootmenu enable='no'/> | </os> | <features> | <acpi/> | <apic/> | <pae/> | </features> | <clock offset='utc'/> | <on_poweroff>destroy</on_poweroff> | <on_reboot>restart</on_reboot> | <on_crash>restart</on_crash> | <devices> | <emulator>/usr/bin/kvm</emulator> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-os'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vda' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage0'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdb' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage1'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdc' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage2'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdd' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage3'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vde' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage4'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdf' bus='virtio' /> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage5'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdg' bus='virtio' /> | </disk> | <interface type='bridge'> | <source bridge='ovsbr0'/> | <virtualport type='openvswitch'> | </virtualport> | <model type='virtio'/> | </interface> | | <controller type='usb' index='0'> | </controller> | <controller type='ide' index='0'> | </controller> | <controller type='virtio-serial' index='0'> | </controller> | <serial type='pty'> | <target port='0'/> | </serial> | <console type='pty'> | <target type='serial' port='0'/> | </console> | <input type='mouse' bus='ps2'/> | <graphics type='vnc' port='-1' autoport='yes'/> | <sound model='ich6'> | </sound> | | <video> | <model type='cirrus' vram='9216' heads='1'/> | </video> | | <memballoon model='virtio'> | </memballoon> | | </devices> | </domain> ##### ---------------------------------------------------------------------------------------------------------------- ##### | <domain type='kvm' id='169'> | <name>arch_test</name> | <uuid>7c3b44dc-ff91-413a-b1cf-dfbe2480d44e</uuid> | <memory unit='KiB'>4194304</memory> | <currentMemory unit='KiB'>4194304</currentMemory> | <vcpu placement='static'>2</vcpu> | <resource> | <partition>/machine</partition> | </resource> | <os> | <type arch='x86_64' machine='pc-i440fx-1.4'>hvm</type> | <boot dev='hd'/> | <bootmenu enable='no'/> | </os> | <features> | <acpi/> | <apic/> | <pae/> | </features> | <clock offset='utc'/> | <on_poweroff>destroy</on_poweroff> | <on_reboot>restart</on_reboot> | <on_crash>restart</on_crash> | <devices> | <emulator>/usr/bin/kvm</emulator> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-os'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vda' bus='virtio'/> | <alias name='virtio-disk0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage0'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdb' bus='virtio'/> | <alias name='virtio-disk1'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage1'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdc' bus='virtio'/> | <alias name='virtio-disk2'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage2'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdd' bus='virtio'/> | <alias name='virtio-disk3'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage3'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vde' bus='virtio'/> | <alias name='virtio-disk4'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage4'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdf' bus='virtio'/> | <alias name='virtio-disk5'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> | </disk> | <disk type='network' device='disk'> | <driver name='qemu'/> | <source protocol='rbd' name='libvirt-pool/arch_test-storage5'> | <host name='192.168.0.35' port='6789'/> | <host name='192.168.0.2' port='6789'/> | <host name='192.168.0.40' port='6789'/> | </source> | <target dev='vdg' bus='virtio'/> | <alias name='virtio-disk6'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> | </disk> | <controller type='usb' index='0'> | <alias name='usb0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> | </controller> | <controller type='ide' index='0'> | <alias name='ide0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> | </controller> | <controller type='virtio-serial' index='0'> | <alias name='virtio-serial0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> | </controller> | <controller type='pci' index='0' model='pci-root'> | <alias name='pci0'/> | </controller> | <interface type='bridge'> | <mac address='52:54:00:d0:da:73'/> | <source bridge='ovsbr0'/> | <virtualport type='openvswitch'> | <parameters interfaceid='f1c41c6e-8ca0-4c34-b1df-053c9a7976bb'/> | </virtualport> | <target dev='vnet2'/> | <model type='virtio'/> | <alias name='net0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> | </interface> | <serial type='pty'> | <source path='/dev/pts/4'/> | <target port='0'/> | <alias name='serial0'/> | </serial> | <console type='pty' tty='/dev/pts/4'> | <source path='/dev/pts/4'/> | <target type='serial' port='0'/> | <alias name='serial0'/> | </console> | <input type='mouse' bus='ps2'/> | <graphics type='vnc' port='5902' autoport='yes' listen='127.0.0.1'> | <listen type='address' address='127.0.0.1'/> | </graphics> | <sound model='ich6'> | <alias name='sound0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> | </sound> | <video> | <model type='cirrus' vram='9216' heads='1'/> | <alias name='video0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> | </video> | <memballoon model='virtio'> | <alias name='balloon0'/> | <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> | </memballoon> | </devices> | <seclabel type='dynamic' model='apparmor' relabel='yes'> | <label>libvirt-7c3b44dc-ff91-413a-b1cf-dfbe2480d44e</label> | <imagelabel>libvirt-7c3b44dc-ff91-413a-b1cf-dfbe2480d44e</imagelabel> | </seclabel> | </domain>