[libvirt-users] Ordering multiple network interfaces on vms
On a hypervisor with multiple interfaces, we'd like to make those interfaces available to each of multiple VMs. We have both bridged interfaces and SRIOV interfaces, no NAT interfaces. When we create a vm in virsh using an xml file with no network elements, but having interface elements, we get the interfaces in the vm, but the ordering of eth devices does not match the ordering on the hypervisor and we aren't sure the ordering is repeatable either. In other words, eth0 in the HV is thingA and eth1 is thingB, but on the VM eth0 is thingC eth1 is thingD and eth4 is thingA so forth. we have 12 interfaces so ordering or naming is important and confusing. We've tried to use the PCI bus numbers as seen in lspci on the HV, but no luck. Do we need to start using network elements in the vm's xml file or separately to control the ordering? Whats the usual practice here? Sample xml: <interface type='hostdev'> <source dev='eth3' mode='vepa'> <address type='pci' domain='0x0000' bus='0x16' slot='0x10' function='0x1'/> </source> <target dev='macvtap0'/> </interface> <interface type='bridge'> <source bridge='br4'/> <target dev='vnet0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> Thanks! -- John Fisher
@Gao see the other thread Managing eths in vm from hv too. We weren't able to order based on vnets, though I agree its seems intuitive. But our test wasn't thorough. Eric says: "Libvirt passes the devices to the qemu command line in the same order that they appear in the XML. It is not libvirt's fault if qemu then takes the command line arguments and presents it to the guest in such a way that the guest can initialize the devices in the order in which a bus scan finds them, rather than in the order they were presented on the command line." as far as I can tell the bridges are always setup first, then the direct VF connections. Whether this is caused by our hardware or Qemu or PCI I have no idea. Now, I am trying the idea of editing the /etc/udev/rules.d/70* file to move the MAC addresses around. Results are mixed. very puzzling. John On 12/08/2012 08:21 AM, Gao Yongwei wrote:
2012/12/7 john fisher <john@jpfisher.net <mailto:john@jpfisher.net>>
<target dev='vnet0'/>
I think the vnet* can affect the order in vm.
-- John Fisher
Gao you were right! see below.... On 12/08/2012 08:21 AM, Gao Yongwei wrote:
2012/12/7 john fisher <john@jpfisher.net <mailto:john@jpfisher.net>>
<target dev='vnet0'/>
I think the vnet* can affect the order in vm.
For the benefit of 1) others and 2) me when I forget how this works- I did find a solution in formatting the xml file. If you leave the vnets out completely, see file below, the generic xml file will cooperate with libvirt and qemu and order the VM's eth devices as they are ordered on the hypervisor. (note: the macvtap entries seen below may also not be needed, sound and usb not tested) ## sample xml file for libvirt 1.0.0 showing some bridges and some SRIOV ports too ## <domain type='kvm' id='1'> <name>sample</name> <hostname>sample</hostname> <memory unit='KiB'>524288</memory> <currentMemory unit='KiB'>524288</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='pc-1.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/oa4-vm-sample-cli.qcow2'/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <controller type='usb' index='0'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <interface type='bridge'> <source bridge='br4'/> <model type='virtio'/> </interface> <interface type='bridge'> <source bridge='br5'/> <model type='virtio'/> </interface> <interface type='bridge'> <source bridge='br6'/> <model type='virtio'/> </interface> <interface type='bridge'> <source bridge='br7'/> <model type='virtio'/> </interface> <interface type='bridge'> <source bridge='br8'/> <model type='virtio'/> </interface> <interface type='bridge'> <source bridge='br9'/> <model type='virtio'/> </interface> <interface type='bridge'> <source bridge='br10'/> <model type='virtio'/> </interface> <interface type='bridge'> <source bridge='br11'/> <model type='virtio'/> </interface> <interface type='bridge'> <source bridge='br250'/> <model type='virtio'/> </interface> <interface type='hostdev'> <source dev='eth0' mode='vepa'> <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x0'/> </source> <target dev='macvtap1'/> <model type='virtio'/> </interface> <interface type='hostdev'> <source dev='eth1' mode='vepa'> <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x1'/> </source> <target dev='macvtap1'/> <model type='virtio'/> </interface> <interface type='hostdev'> <source dev='eth2' mode='vepa'> <address type='pci' domain='0x0000' bus='0x16' slot='0x10' function='0x0'/> </source> <target dev='macvtap0'/> </interface> <interface type='hostdev'> <source dev='eth3' mode='vepa'> <address type='pci' domain='0x0000' bus='0x16' slot='0x10' function='0x1'/> </source> <target dev='macvtap0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <sound model='ich6'> <alias name='sound0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </sound> <video> <model type='cirrus' vram='9216' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> <seclabel type='none'/> </domain> -- John Fisher
participants (2)
-
Gao Yongwei -
john fisher