On 18/11/16 19:17, Andrea Bolognani wrote: If libvirt is using just "pseries" (without a version), then having a virtual PCIe-PCI bridge (and "pci.0" always available by default) will do it. If libvirt is using a specific version of pseries, then it already knows that <=2.7 has pci.0 as a root, pcie.0 otherwise. libvirt has a knowledge what QEMU version has what, right? In what scenario will an additional machine type help? I do not know about i440<->q35 but in pseries the difference is going to be very simple. For example, we did not change the machine type when we switched from default OHCI to XHCI, switching from PCI to PCIe does not sound like we need a whole new machine type for this either. I could do this with the existing pseries if the machine had a "root but type" property. -- Alexey

On 22/11/16 00:08, Andrea Bolognani wrote: Ok. Always likes this approach really. We should start moving to this direction with PHB - stop adding the default PHB at all when -nodefaults is passed (or -machine pseries,pci=off ?) and let libvirt manage PHBs itself (and provide another spapr-phb type like spapr-pcie-host-bridge or add a "pcie_root_bus_type" property to the existing PHB type). What will be wrong with this approach? -- Alexey

On Wed, 2016-11-23 at 16:02 +1100, David Gibson wrote: Note that libvirt always runs QEMU with -nodefaults, so we could just remove the default PHB if that flag is present, and leave it in if that's not the case. The idea itself sounds good, but of course it will require more work from the libvirt side than simply making the PCIe machine type behave like q35 and mach-virt. Moreover, we already have an RFE for supporting additional PHBs, we could solve both issues in one fell swoop and have the libvirt guest XML be a more faithful representation of the actual virtual hardware, which is always a win in my book. That will be even more important if it turns out that the rules for PCIe device assignment (eg. what device/controller can be plugged into which slot) are different for PCIe PHBs than they are for q35/mach-virt PCIe Root Bus. I've asked for clarifications about this elsewhere in the thread. --  Andrea Bolognani / Red Hat / Virtualization

On 11/21/2016 07:08 AM, Andrea Bolognani wrote: But that's okay. In general, we try to promise that: old qemu + old libvirt works new qemu + new libvirt works old qemu + new libvirt works new qemu + old libvirt is untested, and may require a libvirt upgrade In other words, as long as you update your stack from top down (libvirt before qemu), you should be fine. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

On Wed, 2016-11-23 at 16:00 +1100, David Gibson wrote: So, would the PCIe Root Bus in a pseries guest behave differently than the one in a q35 or mach-virt guest? Does it have a different number of slots, do we have to plug different controllers into them, ...? Regardless of how we decide to move forward with the PCIe-enabled pseries machine type, libvirt will have to know about this so it can behave appropriately. Calling the PCIe machine type either pseries-2.8 or pseries-pcie-2.8 would result in the very same amount of work, and in both cases it would be understood that the legacy PCI machine type is no longer going to be updated, but can still be used to run existing guests. --  Andrea Bolognani / Red Hat / Virtualization

On Fri, 2016-12-02 at 15:18 +1100, David Gibson wrote: It's also somewhat broken if they aren't in the same iommu domain because the way we pass the iommu buid is via the parent node, so a given iommu domain must reside below a common parent and not share it.

On Fri, 2016-12-02 at 16:50 +1100, David Gibson wrote: Ok so the H_PUT_TCE calls affect all the underlying tables ? Cheers, Ben.

On 12/01/2016 11:18 PM, David Gibson wrote: That's a broad statement. Why? If qemu reports the default devices and characteristics of the devices properly (and libvirt uses that information) there's no reason for it to make the wrong decision. Explain "kinda-sorta-but-not-really". If there's a deficiency in the model maybe it can be fixed. True, but not for the reasons you think. If qemu is able to respond to queries with adequate details about the devices available for a machinetype (and what buses are in place by default), there's no reason that libvirt can't add devices addressed such that all the connections are legal; what libvirt *can't* get right is the policy requested in the next higher layer of management (and ultimately of the user) - does this device need to be hotpluggable? Does the user want to keep all devices on the root complex to avoid extra PCI controllers? And qemu fundamentally CAN NOT get it right either. qemu knows what is possible and what is allowed, but it doesn't know what the user *wants* (beyond "they want device X", which is only 1/2 the story), and has no way of being told what the user wants other than with a PCI address. To back up for a minute, some background info: once a device has been added to a domain, at *any* time in the future (not just during a migration, but forever more until the end of time) that device must always have the same PCI address as it had that first time. In order to guarantee that, libvirt needs to either: a) keep track of the order the devices were added and always put the devices in the same order on the commandline (assuming that qemu guarantees that it actually assigns addresses based on the order of the devices' appearance on the commandline, which has never been stated anywhere as an API guarantee of qemu), or b) remember the address of each device as it is added and specify that on the commandline in the future. libvirt chooses (b). And where is the logical place to store that address? In the config. So we've established that the PCI address of a device needs to be stored in the config. So why does libvirt need to choose it the first time? 1) Because qemu doesn't have (and CAN NOT have) all the information about what are the user's plans for that device: a) It has no idea if the user wants that device to be hotpluggable (on a root-port) or not (on root complex as an integrated device) b) it doesn't know if the user wants the device to be placed on an expander bus so that its NUMA status can be discovered by the guest OS. If there is a choice, there must be a way to make that choice. The way that qemu provides to make the choice is by specifying an address. So libvirt must specify an address in its config. 2) Because qemu is unable/unwilling to automatically add PCIe root ports when necessary, it's *not even possible* (on PCIe machinetypes) for it to place a device on a hotpluggable port without libvirt specifying a PCI address the very first time the device is added (and also adding in a root-port), but libvirt's default policy is that (almost) all devices should be hotpluggable. If we were to follow your recommendation ("libvirt never specifies PCI addresses, but instead allows qemu to assign them"), hotplug on PCIe-based machinetypes would not be possible, though. There have even been mentions that even libvirt is too *low* in the stack to be specifying the PCI address of devices (i.e. that all PCI address decisions should be up to higher level management applications) - I had posted a patch that would allow specifying "hotpluggable='yes|no'" in the XML rather than forcing the call to specify an address, and this was NACKed because it was seen as libvirt dictating policy. (In the end, libvirt *does* dictate a default policy, (it's just that the only way to modify that policy is by manually specifying addresses) - libvirt's default PCI address policy is that (almost) all devices will be assigned an address that makes them hotpluggable, and will not be placed on a non-0 NUMA node. So, in spite of libvirt's effort, in the end we *still* need to expose address configuration to higher level management applications, since they may want to force all devices onto the root complex (e.g. libguestfs, which does it to reduce PCI controller count, and thus startup time) or force certain devices to be on a non-0 NUMA node (e.g. OpenStack when it wants to place a VFIO assigned device on the same NUMA node in the guest as it is in the host). With all of that, I fail to see how it would be at all viable to simply leave PCI address assignment up to qemu. And since qemu knows about them, it should be able to report them. Which is what Eduardo's work is doing. And then libvirt will know about all the constraints in an programmatic manner (rather than the horrible (tedious, error prone) hardcoding of all those details that we've had to suffer with until now). And since no matter how hard qemu might try to come up with a policy for address assignment that will satisfy the needs of 100% of the users 100% of the time, it will fail (because different users have different needs). Because qemu will be unable to properly place all devices all the time, libvirt (and higher level management) will still need to do it. Even in the basic case qemu doesn't provide what libvirt requires as default - that devices be hotpluggable. No, that doesn't work because qemu would in many situations place the devices at the wrong address / on the wrong controller, because there are many possible topologies that are legal, and the user may (for perfectly valid reasons) want something different from what qemu would have chosen. (An example of two differing (and valid) policies - libguestfs needs guests to startup as quickly as possible, and thus wants as few PCI controllers as possible (this makes a noticeable difference in Linux boot time), so it wants all devices to be integrated on the root complex. On the other hand, a generic guest in libvirt should make all devices hotpluggable just in case the user wants to unplug them, so by default it tries to place all devices on a pcie-root-port. You can't support both of these if addressing is all left up to qemu) It may make you feel good to say that, but the facts don't back it up. Any project makes design mistakes, but in the specific case you're discussing here, I think you haven't looked from a wide enough viewpoint to see the necessity of what libvirt is doing and why it can't be done by qemu (certainly not all the time anyway). libvirt has always done the best that could be done with the information provided by qemu. The problem isn't that libvirt is creating new problems for qemu out of thin air, it's that qemu is unable to automatically address PCI devices for all possible situations and user policy preferences, so higher levels need to make the decisions about addressing to satisfy their policies (ie what they *want*, eg hotpluggable, integrated on root complex), and qemu hasn't (until Eduardo's patches) been able to provide adequate information about what is *legal* (e.g which devices can be plugged into which model of pci controller, what slots are available on each type of controller, whether those slots are hotpluggable) in a programmatic way, so libvirt has had to hardcode rules about bus-device compatibility and capabilities, slot ranges, etc in order to make proper decisions itself when possible, and to sanity-check decisions about addresses made by higher level management when not. I don't think that's a design flaw. I think that's making the best of a "less than ideal" situation. Eduardo's work isn't being done to make up for some mythical design flaw in libvirt. It is being done in order to give libvirt the (previously unavailable) information it needs to do a necessary job, and is being done at least partly at the request of libvirt (we've certainly been demanding some of that stuff for a long time!) The summary is that it's impossible for qemu to correctly decide where to put new devices, especially in a PCIe hierarchy for a few reasons (at least); because of this, libvirt (and higher level management) needs to be able to assign addresses to devices, and in order for us/them to be able to do that properly, qemu needs to provide detailed and accurate information about what buses/controllers/devices are in each machinetype, what controllers/devices are available to add, and what are the legal ways of connecting those devices and controllers.

On Mon, Dec 05, 2016 at 03:54:49PM -0500, Laine Stump wrote: Actually, unlike the comments below, this assertion is neither particularly broad, nor critical of libvirt (it is obliquely critical of PAPR and PowerVM). The point is that the way PCI-E devices are presented under PAPR is different from the way they'd appear on a hardware bus or on virtualized PC or ARM. Specifically we don't present the pseudo-P2P bridges of the root complexes to the guest. Instead (unless you explicitly ask for a bridge) all the devices will appear directly on the root "bus" - obviously this bus is a fiction since PCI-E is point to point in hardware. In short, from the point of view of topology, PAPR PCI-E will look much more like vanilla PCI than PCI-E on an x86. So to the extent that libvirt address allocation differs between PCI and PCI-E, and I gather it does, the PCI-E variant will be wrong for PAPR. That's why for the medium term I advocate option (1) above of treating PAPR PCI-E as if it was a vanilla PCI bus, but with a flag indicating that PCI-E devices can be attached to it. My point is that the variety of constraints we can have in the back end is so broad that it's impossible to construct a representation to that level of detail which is truly backend independent. The problem here is "adequate details". Again, AFAICT the variety of possible constraints given all the vagaries of different platforms is so great that I don't think any format short of code is sufficient to describe it. So, here's a more specific point. You *don't* give PCI addresses to qemu - rather you give device/slot numbers, and which bus/bridge they're attached to. libvirt's use of DDDD:BB:SS.F addresses is a fundamental error, because such addresses aren't meaningful until *after* the bus is enumerated. The bus number is explicitly allocated during enumeration (by guest firmware or OS). You can kind of predict what they'll do, but it requires knowing about every bridge and device on the system, which means it risks breakage every time the backend changes its default setup - and the interpretation of the addresses could be changed by an insertion of a bridge somewhere apparently unrelated in the XML. The domain number is an entirely guest-local concept, except - sorta - on x86 where there's an obvious choice of the index that's used to control the multiplexed access to all the host bridges' config spaces. On platforms which have separate registers for each host bridge, it's entirely possible for libvirt's idea of domain numbers to differ from the guest's which is wonderfully confusing. I dispute the "any time". I'll grant you want to keep the virtual hardware stable for an extended period. But if you ever want to update your backend to newer behaviour (meaning new machine type version, not just new qemu version) then you should revisit address allocation. And, yes, that potentially breaks the guest, but so could the backend changes anyway, just like any hardware update for a physical machine. In most case - assuming it's been set up sensibly - the guest OS will cope with minor hw reshuffles. That is, after all, why RHEL locates filesystems by UUID and NICs by MAC in normal operation. Putting in the config seems like the logical place at first, but it's a mistake: First, the information we need to ensure stable guest hardware is inherently backend specific. On qemu we need to say which bridges and bus ids each device is connected to. Some other system could use PCI addresses (I think that would be a a design flaw in the backend, but if libvirt can't deal with imperfect backends it has failed its remit), for containers the whole question doesn't even make sense. A really simplistic hypervisor could take simply an (ordered) list of plugin devices and always put them into a fixed set of slots. Putting this into the general config forces us to attempt a solution to the insoluble: greating a neutral representation translatable to any backend specific representation. Second, this means that the config now contains both information that was originally entered by the user (or next level up) as what they want for the VM, and also information that libvirt decided once upon a time and is now sticking with. We can no longer separate which bits of information are which. That means when we want to update the backend, we can't tell what we should regenerate. [Meta rant, when it comes to design flaws that breed other design flaws the fact that the domain XML is both a read and write interface is just about the granddaddy of them all. It has two awful consequences: 1) After the VM is first run, we can no longer tell which config constraints were ones actually requested by the user, and which were added by some version of libvirt in the past and are now just there so we make the same decisions as before. That distinction matters. 2) Because the user could be relying on *seeing* something in the XML, not just that whatever it put in there will still be understood by new libvirt versions means we basically can't ever fix errors in the XML design.] CAN not? Why not? Fundamentally it seems like we need to get the information about what the user wants and what the virtual hardware can do into the same place. At least from the examples you've mentioned so far, pushing "needs to be hotpluggable" and so forth flags down into qemu seems a lot more feasible that pulling up the complex and platform dependent set of constraints into libvirt. And yes, qemu is very far from perfect in this area - the introspection interfaces need to be greatly improved. So both (a) and (b) are precisely examples of what I'm describing. libvirt is assuming (a) requires hotplug => must be on a root port (b) device bound to NUMA node => use expander bus But (a) is only true for guest platforms using the standard hotplug mechanism. pseries does not. (b) is essentially only true for x86 - the whole concept of the expander bus is a hack to workaround x86's historical inability to have truly independent PCI host bridge. On pseries we can create independent host bridges and assign them each a NUMA node (including the default one). qemu already knew that, libvirt needs to be taught. Just as it seems like libvirt needs to be taught about practically every platform quirk. Again, no, qemu does not take addresses at all. At least not in the "DDDD:BB:SS.F" sense. Even if it did, DDDD:BB:SS.F addresses have no place in a supposedly back-end independent config. Ok, so, yes, I see that a lot of this allocation mess originates because qemu didn't provide sufficient auto-allocation, or introspection, so libvirt had to work around that. The trouble is that those workarounds have now been baked into the libvirt interface - primarily by the fact that the data is integrated back into the primary config rather than stored as adjunct back end specific data. That means it's now insanely difficult to improve the interfaces to qemu, because libvirt XMLs now have all this address information. A few of them are because the user really did request that for some reason, but many are because that was how libvirt tried to implement some more abstract user request. We can now no longer tell the difference, which means we can't translate those abstract requests into a new saner qemu interface. Frankly, if the higher level management needs that much control over the PCI topology, going via an abstraction layer like libvirt seems not the right choice. In that case the management needs to understand the guest platform in great detail anyway, so what does libvirt buy you. Huh. Your patch makes perfect sense to me - a step in the right direction - and the reason for NACK seems completely bogus. Yeah, but "address which makes them hotpluggable" depends on guest platform (machine type). As does how you assign them to NUMA nodes. So, I'm not necessarily against allowing an explicit address override for special cases (although DDDD:BB:SS.F is the wrong way to encode it). If that breaks due to a backend update, it's reasonable to just punt the error back to the user and get them to fix it - if they didn't understand the guest platform well enough to fix this, they shouldn't have been overriding addresses. But it's not reasonable to punt back to the user when the only thing that got broken by the backend update was information libvirt (maybe not even the current version) decided some time in the past, based on faulty assumptions about the guest platform. But libvirt has no way to distinguish these cases because all the info is folded back into the one XML blob. I haven't had a chance to look in detail at Eduardo's latest stuff, but yes it should be an improvement. But I think you vastly underestimate how many different platform quirks we could have which won't be easy to encode. Right, there are certainly reasons for overrides. But at the moment *every* device is overridden, which means when there is something wrong in libvirt's assumptions about how to place things, every device can break, not just overridden ones. With the tools that qemu currently gives you. But now the assumption that address allocation belongs to libvirt means that it's insanely difficult to build those tools without breaking libvirt. Well, I did warn you it was a rant. That said, while there are understandable historical reasons for them design flaws in libvirt which make it really difficult to design other things properly are alas not a myth. Three examples: 1) Read/write XML interface. Makes it nigh-impossible to fix flaws by updating to a more coherent internal representation (with backwards compat glue to parse the old format into the new one). 2) Not separating original user config from libvirt additions / annotations. Discussed repeatedly above. 3) Use of DDDD:BB.SS.F addresses (rather than a reference to bus attachment point + SS.F). You can't use addresses to construct a machine in an address space which only exists once the machine is constructed. And that's just the first three I came up with that have at least a slight connection to the problem at hand. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson

On Tue, Dec 06, 2016 at 06:30:47PM +0100, Andrea Bolognani wrote: Well, in a sense that's up to us. The only existing model we have is PowerVM, and PowerVM only does device passthrough, no emulated devices. PAPR doesn't really distinguish one way or the other, but it's written from the perspective of assuming that all PCI devices correspond to physical devices on the host Right, my understanding was that if the devices were slotted, rather than integrated, each one would sit under a separate root complex, the root complex being a pseudo PCI to PCI bridge. The qemu default, or the libvirt default? I think this represents treating the devices as though they were integrated devices in the host bridge. I believe on q35 they would not be hotpluggable - but on pseries they would be (because we don't use the standard hot plug controller). Uh.. AIUI the root complex is the PCI to PCI bridge under which PCI-E slots appear. PXB is something different - essentially different host bridges as you say (though with some weird hacks to access config space, which make it dependent on the primary bus in a way which spapr PHBs are not). I'll admit I'm pretty confused myself about the exact distinction between root complex, root port and upstream and downstream ports. On pseries it won't. But my understanding is that libvirt won't create them that way on q35 - instead it will insert the RCs / P2P bridges to allow them to be hotplugged. Inserting that bridge may confuse pseries guests which aren't expecting it. Here you've hit on it. No, we should not do that for pseries, AFAICT. PAPR doesn't really have the concept of integrated endpoint devices, and all devices can be hotplugged via the PAPR mechanisms (and none can via the PCI-E standard hotplug mechanism). I wish it were entirely clear in my mind. Like I say I'm still pretty confused by exactly the root complex entails. That we'd have to address on the qemu side with some Like I say, I'm fairly confused myself, but I'm pretty sure that Root Complex != Root Bus. The RC sits under the root bus IIRC.. or possibly it consists of the root bus plus something under it as well. Now... from what Laine was saying it sounds like more of the differences between PCI-E placement and PCI placement may be implemented by libvirt than qemu than I realized. So possibly we do want to make the bus be PCI-E on the qemu side, but have libvirt use the vanilla-PCI placement guidelines rather than PCI-E for pseries. Hrm, the suggestion of providing both a vanilla-PCI and PCI-E host bridge came up before. I think one of us spotted a problem with that, but I don't recall what it was now. I guess one is how libvirt would map it's stupid-fake-domain-numbers to which root bus to use. Hrm. So my first inclination would be to stick with the generic names, and map those to creating new pseries host bridges on pseries guests. I would have thought that would be the easier option for users. But I may not have realized all the implications yet. I think the above I've touched on in my reply to Laine. Right.. so here's the thing, I strongly suspect that Marcel's guidelines will not be correct for pseries. I'm not sure if they'll be definitively wrong, or just different enough from PowerVM that it might confuse guests, but either way. Can you send me a link to that document though, which might help me figure this out. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson

On 12/07/2016 06:42 PM, Andrea Bolognani wrote: Hi, Sorry for the late reply. Not a PCIe Root Complex, but a PCIe Root port. Correct. Please have a look to the new document regarding pcie: docs/pcie.txt and the corresponding presentations. right right Same applies pxb should not be used for PCIe machines, only for legacy PCI ones. OK I agree If we had spapr-pcie-host-bridge, we Actually the root PCIe bus is part of a root complex. But I suspect that's likely to cause This seems to be interfering with the PCIe spec: 1. No PCIe root ports ? those are part of the spec. 2. Only integrated devices ? hotplug is not PCIe native? The Root complex includes the PCI bus, some configuration registers if needed, provides access to the configuration space, translates relevant CPU reads/writes to PCI(e) transactions... That looks complicated.. I wish I would no more about the pseries PCIe stuff, does any one know where I can get the info ? (besides 'google it'...) This would be a weird configuration, I never heard of something like that on a bare metal machine, but I never worked on pseries, who knows... Indeed, all the pxb-pcie devices "share" the same domain. We should make the document stick for all PCIe archs, if we need to modify it, let's do it. I'm not sure if they'll I really need to understand how it would confuse the guests, it does not deviate from the PCIe spec, it only adds some restrictions. I kind of miss the core issue, what is the main problem? Thanks, Marcel

On Tue, 2016-12-13 at 14:25 +0200, Marcel Apfelbaum wrote: That's the thing though, it's *not* bare metal ;-) On bare metal, we use the "powernv" platform for which we haven't yet upstreamed the PCIE support, but there we have real PCIe root ports with all what's exepcted on them. They come on physically different PHB, one root complex per PHB, but they are "proper" PCIe. Hotplug is a bit weird still because it has to go through some FW interactions as the HW doesn't use the PCIe standard slot control bits (and our qemu model doesn't handle it yet) but otherwise it's standard. "pseries" on the other hand is a paravirtualized platform. It's the environment presented to guests by our propritary hypervisor (PowerVM, aka pHyp) and KVM/qemu. I think there's a public spec these days but to cut the story short, it doesn't expose PCIe "properly" for bad historical reasons. It tends to hide the parent, ie, the root port for slots connected directly to a PHB or the entire hierarchy from the root to (and including) the downstream switch port for slots under as switch. So you end up with PCIe devices devoid of a proper "parent" which is a bit weird. Hotplug is implemented using specific firmware interfaces that are identical with pre-PCIe (ie PCI/PCI-X) systems. The FW has interface for supporting 3 kinds of hotplug: - Entire PHBs - P2P Bridges - Individual devices on an existing PHB or P2P bridge In practice these days mostly the first one is used for everything under pHyp, though I think we have a mix of 1 and 3 for KVM/qemu. Cheers, Ben.

On 12/13/2016 05:15 PM, Benjamin Herrenschmidt wrote: Thanks for the explanations, so bare metal is standard minus hotplug. OK, now is more "clear". I can get back to read the thread from the start :) Thanks, Marcel

On 12/14/2016 04:46 AM, David Gibson wrote: Yes, a Root Complex is a type of Host Bridge in the sense it bridges between CPU/Memory Controller ant the PCI subsystem. OK Yes The point is that Understood, on one hand we have no PCIe Root Ports, on the other hand the devices are not integrated - they can be hot-plugged by platform specific means. Ok the hw is PCIe, but configuration/hot-plug is platform specific. Root Complexes use MMIO to expose the PCI configuration space, they call it ECAM (enhanced configuration access mechanism) or MMConfig. Understood, I'll have a look, but with low expectations :). Anyway, by now I do have some basic notions of spapr, thanks! I agree about the multiple host-bridges, is actually what pxb/pxb-pcie devices (kind of) do. I was talking about having one PCI PHB and another PHB which is PCI Express. I was referring to one system having both PCI and PCIe PHBs. Understood. At this point I agree that PAPR PCI-E is not really "by the book", so the PCIe guidelines will not work for PAPR. A PCIe bus supporting PCI devices is strange (QEMU allows it ...), but a PCI bus supporting PCIe devices is hard to "swallow". I would say maybe make it a special case of a PCIe bus with different rules. It can derive from the PCIe bus class and override the usual behavior with PAPR specific rule which happen to be similar with the PCI bus rules. Adding Eduardo, he is currently working on a way to properly expose the information on what devices can be plugged on what bus/slot. Do we have in QEMU the code to expose the Extended Config Space by other means instead of MMIO (used by x86)? So the problem is related to how to expose the information to libvirt? If yes, maybe Eduardo can help. Thanks, Marcel

On Wed, 2016-12-14 at 20:26 +0200, Marcel Apfelbaum wrote: Sorry this confused you: what I was talking about was just having both a legacy PCI PHB and a PCI Express PHB available in QEMU, not using both for the same guest. The user would pick one or the other and stick with it. I agree. It's pretty clear by now that libvirt will need to have some special handling of pseries guests. Having to choose between "weird legacy PCI" and "weird PCI Express", I think I'd rather go with the latter :) --  Andrea Bolognani / Red Hat / Virtualization