[libvirt-users] avoiding PCI bus 8 / using PCI function / virt-install
I’m using virt-install to spin up VMs. At times I have a need to spin up VMs which have 100s of interfaces. I ran into the PCI issue mentioned in this previous thread based on how virt-install assigns PCI addresses to interfaces: https://www.redhat.com/archives/libvirt-users/2018-December/msg00064.html Using the info mention there I was able to part hand / part sw re-write an XML where I would remove PCI bus references above the value of 8 and re-address the per interface PCI info to use the function field (I don’t need hot pluggable). But the process I’ve built is brittle. Wondering what options i might have to better deal with this PCI issue. Is there a way to tell virt-install, when building the info it passes to qemu, to use the function field during the PCI assignment process so as to support many more interfaces before hitting the PCI bus == 8 issue? If not, is there a way with one of the virt command line tools to create the XML (with the PCI addresses specified) so that I can process that XML and re-write the PCI addressing values? Right now the only way I’ve been able to get that detailed XML file is to 1) virt-install and let the VM begin the boot process and then do a 2) virsh dumpxml and then 3) virsh destroy/undefine that VM, 4) modify the XML and then 5) virsh create ./modified.xml. Is there a cleaner way to do this? Thanks
On Wed, 26 Dec 2018 at 16:26, b f31415 <bf31415@gmail.com> wrote:
If not, is there a way with one of the virt command line tools to create the XML (with the PCI addresses specified) so that I can process that XML and re-write the PCI addressing values? Right now the only way I’ve been able to get that detailed XML file is to 1) virt-install and let the VM begin the boot process and then do a 2) virsh dumpxml and then 3) virsh destroy/undefine that VM, 4) modify the XML and then 5) virsh create ./modified.xml. Is there a cleaner way to do this?
It won't do everything you want, but check out virt-install --print-step --dry-run. This should at least allow you to get at the XML for the intermediate steps and remove the brittleness of defining, destroying, undefining the VM. Cheers, - Peter
Thanks. Maybe I'm missing something but when I try virt-install --dry-run --print-xml it generates this snippet of XML: <interface type="bridge"> <source bridge="ge_0_0_0_77"/> <mac address="52:54:00:01:00:4e"/> <model type="virtio"/> </interface> but what i need (and get from virsh dumpxml) has this extra PCI line. <interface type='bridge'> <mac address='52:54:00:01:00:4e'/> <source bridge='ge_0_0_0_77'/> <target dev='vnet78'/> <model type='virtio'/> <alias name='net78'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x0a' function='0x6' /> </interface> Did I miss something? Do you see PCI in the XML when you do the virt-install --dryrun approach? Thanks On Wed, Dec 26, 2018 at 2:57 PM Peter Crowther <peter.crowther@melandra.com> wrote:
On Wed, 26 Dec 2018 at 16:26, b f31415 <bf31415@gmail.com> wrote:
If not, is there a way with one of the virt command line tools to create the XML (with the PCI addresses specified) so that I can process that XML and re-write the PCI addressing values? Right now the only way I’ve been able to get that detailed XML file is to 1) virt-install and let the VM begin the boot process and then do a 2) virsh dumpxml and then 3) virsh destroy/undefine that VM, 4) modify the XML and then 5) virsh create ./modified.xml. Is there a cleaner way to do this?
It won't do everything you want, but check out virt-install --print-step --dry-run. This should at least allow you to get at the XML for the intermediate steps and remove the brittleness of defining, destroying, undefining the VM.
Cheers,
- Peter
On 12/26/18 6:59 PM, b f31415 wrote:
Thanks. Maybe I'm missing something but when I try virt-install --dry-run --print-xml it generates this snippet of XML:
<interface type="bridge">
<source bridge="ge_0_0_0_77"/>
<mac address="52:54:00:01:00:4e"/>
<model type="virtio"/>
</interface>
but what i need (and get from virsh dumpxml) has this extra PCI line.
<interface type='bridge'>
<mac address='52:54:00:01:00:4e'/>
<source bridge='ge_0_0_0_77'/>
<target dev='vnet78'/>
<model type='virtio'/>
<alias name='net78'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x0a' function='0x6' />
</interface>
Did I miss something? Do you see PCI in the XML when you do the virt-install --dryrun approach?
virt-install never explicitly specifies a PCI address in the XML it creates and sends to libvirt, and what you're seeing as the output from --print-xml is the XML before it has been sent to libvirt. Additionally, in response to a question in the original mail, there is no way to tell virt-install (or libvirt itself) to use non-0 functions for devices (except to manually specify the entire PCI address). Since 99 44/100% of users (probably much higher, but that number has such a nice ring to anyone who recalls the old Ivory Soap commercials :-) never need more devices than are possible using only function 0, there has been no effort made to automate assignment of non-0 functions. So if you need this many devices, you should probably consider just using virt-install once to generate proto-template XML, then "virsh define" that template to get an XML that has an <address...> element for every device that needs it, write a script that condenses all the device PCI addresses into less slots by using non-0 functions (and saves that as the "final" template), and finally modify your installation to do a series of virsh commands using that template XML as a start. I know it sounds awkward, but there is really nothing in libvirt or virt-install to automate assignment of PCI devices to non-0 functions.
On Wed, Dec 26, 2018 at 2:57 PM Peter Crowther <peter.crowther@melandra.com <mailto:peter.crowther@melandra.com>> wrote:
On Wed, 26 Dec 2018 at 16:26, b f31415 <bf31415@gmail.com <mailto:bf31415@gmail.com>> wrote:
If not, is there a way with one of the virt command line tools to create the XML (with the PCI addresses specified) so that I can process that XML and re-write the PCI addressing values? Right now the only way I’ve been able to get that detailed XML file is to 1) virt-install and let the VM begin the boot process and then do a 2) virsh dumpxml and then 3) virsh destroy/undefine that VM, 4) modify the XML and then 5) virsh create ./modified.xml. Is there a cleaner way to do this?
It won't do everything you want, but check out virt-install --print-step --dry-run. This should at least allow you to get at the XML for the intermediate steps and remove the brittleness of defining, destroying, undefining the VM.
Cheers,
- Peter
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Thanks-- i get this eco-system setup isn't mainstream (yet). just wanted to double check to see if there was some pre-existing way to use the function aspect. B On Wed, Jan 2, 2019 at 9:29 AM Laine Stump <laine@redhat.com> wrote:
On 12/26/18 6:59 PM, b f31415 wrote:
Thanks. Maybe I'm missing something but when I try virt-install --dry-run --print-xml it generates this snippet of XML:
<interface type="bridge">
<source bridge="ge_0_0_0_77"/>
<mac address="52:54:00:01:00:4e"/>
<model type="virtio"/>
</interface>
but what i need (and get from virsh dumpxml) has this extra PCI line.
<interface type='bridge'>
<mac address='52:54:00:01:00:4e'/>
<source bridge='ge_0_0_0_77'/>
<target dev='vnet78'/>
<model type='virtio'/>
<alias name='net78'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x0a' function='0x6' />
</interface>
Did I miss something? Do you see PCI in the XML when you do the virt-install --dryrun approach?
virt-install never explicitly specifies a PCI address in the XML it creates and sends to libvirt, and what you're seeing as the output from --print-xml is the XML before it has been sent to libvirt.
Additionally, in response to a question in the original mail, there is no way to tell virt-install (or libvirt itself) to use non-0 functions for devices (except to manually specify the entire PCI address).
Since 99 44/100% of users (probably much higher, but that number has such a nice ring to anyone who recalls the old Ivory Soap commercials :-) never need more devices than are possible using only function 0, there has been no effort made to automate assignment of non-0 functions.
So if you need this many devices, you should probably consider just using virt-install once to generate proto-template XML, then "virsh define" that template to get an XML that has an <address...> element for every device that needs it, write a script that condenses all the device PCI addresses into less slots by using non-0 functions (and saves that as the "final" template), and finally modify your installation to do a series of virsh commands using that template XML as a start.
I know it sounds awkward, but there is really nothing in libvirt or virt-install to automate assignment of PCI devices to non-0 functions.
On Wed, Dec 26, 2018 at 2:57 PM Peter Crowther <peter.crowther@melandra.com <mailto:peter.crowther@melandra.com>> wrote:
On Wed, 26 Dec 2018 at 16:26, b f31415 <bf31415@gmail.com <mailto:bf31415@gmail.com>> wrote:
If not, is there a way with one of the virt command line tools to create the XML (with the PCI addresses specified) so that I can process that XML and re-write the PCI addressing values? Right now the only way I’ve been able to get that detailed XML file is to 1) virt-install and let the VM begin the boot process and then do a 2) virsh dumpxml and then 3) virsh destroy/undefine that VM, 4) modify the XML and then 5) virsh create ./modified.xml. Is there a cleaner way to do this?
It won't do everything you want, but check out virt-install --print-step --dry-run. This should at least allow you to get at the XML for the intermediate steps and remove the brittleness of defining, destroying, undefining the VM.
Cheers,
- Peter
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
participants (3)
-
b f31415 -
Laine Stump -
Peter Crowther