1:30 a.m.
On Fri, 2016-12-02 at 15:18 +1100, David Gibson wrote:
> So, would the PCIe Root Bus in a pseries guest behave
> differently than the one in a q35 or mach-virt guest?
Yes. I had a long discussion with BenH and got a somewhat better idea
about this.
Sorry, but I'm afraid you're going to have to break this
down even further for me :(
If only a single host PE (== iommu group) is passed through and
there
are no emulated devices, the difference isn't too bad: basically on
pseries you'll see the subtree that would be below the root complex on
q35.
But if you pass through multiple groups, things get weird.
Is the difference between q35 and pseries guests with
respect to PCIe only relevant when it comes to assigned
devices, or in general? I'm asking this because you seem to
focus entirely on assigned devices.
On q35,
you'd generally expect physically separate (different slot) devices to
appear under separate root complexes.
This part I don't get at all, so please bear with me.
The way I read it you're claiming that eg. a SCSI controller
and a network adapter, being physically separate and assigned
to separate PCI slots, should have a dedicated PCIe Root
Complex each on a q35 guest.
That doesn't match with my experience, where you would simply
assign them to separate slots of the default PCIe Root Bus
(pcie.0), eg. 00:01.0 and 00:02.0.
Maybe you're referring to the fact that you might want to
create multiple PCIe Root Complexes in order to assign the
host devices to separate guest NUMA nodes? How is creating
multiple PCIe Root Complexes on q35 using pxb-pcie different
than creating multiple PHBs using spapr-pci-host-bridge on
pseries?
Whereas on pseries they'll
appear as siblings on a virtual bus (which makes no physical sense for
point-to-point PCI-E).
What is the virtual bus in question? Why would it matter
that they're siblings?
I'm possibly missing the point entirely, but so far it
looks to me like there are different configurations you
might want to use depending on your goal, and both q35
and pseries give you comparable tools to achieve such
configurations.
I suppose we could try treating all devices on pseries as though
they
were chipset builtin devices on q35, which will appear on the root
PCI-E bus without root complex. But I suspect that's likely to cause
trouble with hotplug, and it will certainly need different address
allocation from libvirt.
PCIe Integrated Endpoint Devices are not hotpluggable on
q35, that's why libvirt will follow QEMU's PCIe topology
recommendations and place a PCIe Root Port between them;
I assume the same could be done for pseries guests as
soon as QEMU grows support for generic PCIe Root Ports,
something Marcel has already posted patches for.
Again, sorry for clearly misunderstanding your explanation,
but I'm still not seeing the issue here. I'm sure it's very
clear in your mind, but I'm afraid you're going to have to
walk me through it :(
> Regardless of how we decide to move forward with the
> PCIe-enabled pseries machine type, libvirt will have to
> know about this so it can behave appropriately.
So there are kind of two extremes of how to address this. There are a
variety of options in between, but I suspect they're going to be even
more muddled and hideous than the extremes.
1) Give up. You said there's already a flag that says a PCI-E bus is
able to accept vanilla-PCI devices. We add a hack flag that says a
vanilla-PCI bus is able to accept PCI-E devices. We keep address
allocation as it is now - the pseries topology really does resemble
vanilla-PCI much better than it does PCI-E. But, we allow PCI-E
devices, and PAPR has mechanisms for accessing the extended config
space. PCI-E standard hotplug and error reporting will never work,
but PAPR provides its own mechanisms for those, so that should be ok.
We can definitely special-case pseries guests and take
the "anything goes" approach to PCI vs PCIe, but it would
certainly be nicer if we could avoid presenting our users
the head-scratching situation of PCIe devices being plugged
into legacy PCI slots and still showing up as PCIe in the
guest.
What about virtio devices, which present themselves either
as legacy PCI or PCIe depending on the kind of slot they
are plugged into? Would they show up as PCIe or legacy PCI
on a PCIe-enabled pseries guest?
2) Start exposing the PCI-E heirarchy for pseries guests much more
like q35, root complexes and all. It's not clear that PAPR actually
*forbids* exposing the root complex, it just doesn't require it and
that's not what PowerVM does. But.. there are big questions about
whether existing guests will cope with this or not. When you start
adding in multiple passed through devices and particularly virtual
functions as well, things could get very ugly - we might need to
construct multiple emulated virtual root complexes or other messes.
In the short to medium term, I'm thinking option (1) seems pretty
compelling.
Is the Root Complex not currently exposed? The Root Bus
certainly is, otherwise PCI devices won't work at all, I
assume. And I can clearly see a pci.0 bus in the output
of 'info qtree' for a pseries guest, and a pci.1 too if
I add a spapr-pci-host-bridge.
Maybe I just don't quite get the relationship between Root
Complexes and Root Buses, but I guess my question is: what
is preventing us from simply doing whatever a
spapr-pci-host-bridge is doing in order to expose a legacy
PCI Root Bus (pci.*) to the guest, and create a new
spapr-pcie-host-bridge that exposes a PCIe Root Bus (pcie.*)
instead?
So, I'm not sure if the idea of a new machine type has legs or
not,
but let's think it through a bit further. Suppose we have a new
machine type, let's call it 'papr'. I'm thinking it would be (at
least with -nodefaults) basically a super-minimal version of pseries:
so each PHB would have to be explicitly created, the VIO bridge would
have to be explicitly created, likewise the NVRAM. Not sure about the
"devices" which really represent firmware features - the RTC, RNG,
hypervisor event source and so forth.
Might have some advantages. Then again, it doesn't really solve the
specific problem here. It means libvirt (or the user) has to
explicitly choose a PCI or PCI-E PHB to put things on,
libvirt would probably add a
<controller type='pci' model='pcie-root'/>
to the guest XML by default, resulting in a
spapr-pcie-host-bridge providing pcie.0 and the same
controller / address allocation logic as q35; the user
would be able to use
<controller type='pci' model='pci-root'/>
instead to stick with legacy PCI. This would only matter
when using '-nodefaults' anyway, when that flag is not
present a PCIe (or legacy PCI) could be created by QEMU
to make it more convenient for people that are not using
libvirt.
Maybe we should have a different model, specific to
pseries guests, instead, so that all PHBs would look the
same in the guest XML, something like
<controller type='pci' model='phb-pcie'/>
It would require shuffling libvirt's PCI address allocation
code around quite a bit, but it should be doable. And if it
makes life easier for our users, then it's worth it.
but libvirt's
PCI-E address allocation will still be wrong in all probability.
Guh.
As an aside, here's a RANT.
[...]
Laine already addressed your points extensively, but I'd
like to add a few thoughts of my own.
* PCI addresses for libvirt guests don't need to be stable
only when performing migration, but also to guarantee
that no change in guest ABI will happen as a consequence
of eg. a simple power cycle.
* Even if libvirt left all PCI address assignment to QEMU,
we would need a way for users to override QEMU's choices,
because one size never fits all and users have all kinds
of crazy, yet valid, requirements. So the first time we
run QEMU, we would have to take the backend-specific
format you suggest, parse it to extract the PCI addresses
that have been assigned, and reflect them in the guest
XML so that the user can change a bunch of them. Then I
guess we could re-encode it in the backend-specific format
and pass it to QEMU the next time we run it but, at that
point, what's the difference with simply putting the PCI
addresses on the command line directly?
* It's not just about the addresses, by the way, but also
about the controllers - what model is used, how they are
plugged together and so on. More stuff that would have to
round-trip because users need to be able to take matters
into their own hands.
* Design mistakes in any software, combined with strict
backwards compatibility requirements, make it difficult
to make changes in both related components and the
software itself, even when the changes would be very
beneficial. It can be very frustrating at times, but
it's the reality of things and unfortunately there's only
so much we can do about it.
* Eduardo's work, which you mentioned, is going to be very
beneficial in the long run; in the short run, Marcel's
PCIe device placement guidelines, a document that has seen
contributions from QEMU, OVMF and libvirt developers, have
been invaluable to improve libvirt's PCI address allocation
logic. So we're already doing better, and more improvements
are on the way :)
--
Andrea Bolognani / Red Hat / Virtualization