11:46 p.m.
There was a discussion back in November on the qemu list which spilled
onto the libvirt list about how to add support for PCIe devices to
POWER VMs, specifically 'pseries' machine type PAPR guests.
Here's a more concrete proposal for how to handle part of this in
future from the libvirt side. Strictly speaking what I'm suggesting
here isn't intrinsically linked to PCIe: it will make adding PCIe
support sanely easier, as well as having a number of advantages for
both PCIe and plain-PCI devices on PAPR guests.
Background:
* Currently the pseries machine type only supports vanilla PCI
buses.
* This is a qemu limitation, not something inherent - PAPR guests
running under PowerVM (the IBM hypervisor) can use passthrough
PCIe devices (PowerVM doesn't emulate devices though).
* In fact the way PCI access is para-virtalized in PAPR makes the
usual distinctions between PCI and PCIe largely disappear
* Presentation of PCIe devices to PAPR guests is unusual
* Unlike x86 - and other "bare metal" platforms, root ports are
not made visible to the guest. i.e. all devices (typically)
appear as though they were integrated devices on x86
* In terms of topology all devices will appear in a way similar to
a vanilla PCI bus, even PCIe devices
* However PCIe extended config space is accessible
* This means libvirt's usual placement of PCIe devices is not
suitable for PAPR guests
* PAPR has its own hotplug mechanism
* This is used instead of standard PCIe hotplug
* This mechanism works for both PCIe and vanilla-PCI devices
* This can hotplug/unplug devices even without a root port P2P
bridge between it and the root "bus
* Multiple independent host bridges are routine on PAPR
* Unlike PC (where all host bridges have multiplexed access to
configuration space) PCI host bridges (PHBs) are truly
independent for PAPR guests (disjoint MMIO regions in system
address space)
* PowerVM typically presents a separate PHB to the guest for each
host slot passed through
The Proposal:
I suggest that libvirt implement a new default algorithm for placing
(i.e. assigning addresses to) both PCI and PCIe devices for (only)
PAPR guests.
The short summary is that by default it should assign each device to a
separate vPHB, creating vPHBs as necessary.
* For passthrough sometimes a group of host devices can't be safely
isolated from each other - this is known as a (host) Partitionable
Endpoint (PE). In this case, if any device in the PE is passed
through to a guest, the whole PE must be passed through to the
same vPHB in the guest. From the guest POV, each vPHB has exactly
one (guest) PE.
* To allow for hotplugged devices, libvirt should also add a number
of additional, empty vPHBs (the PAPR spec allows for hotplug of
PHBs, but this is not yet implemented in qemu). When hotplugging
a new device (or PE) libvirt should locate a vPHB which doesn't
currently contain anything.
* libvirt should only (automatically) add PHBs - never root ports or
other PCI to PCI bridges
In order to handle migration, the vPHBs will need to be represented in
the domain XML, which will also allow the user to override this
topology if they want.
Advantages:
There are still some details I need to figure out w.r.t. handling PCIe
devices (on both the qemu and libvirt sides). However the fact that
PAPR guests don't typically see PCIe root ports means that the normal
libvirt PCIe allocation scheme won't work. This scheme has several
advantages with or without support for PCIe devices:
* Better performance for 32-bit devices
With multiple devices on a single vPHB they all must share a (fairly
small) 32-bit DMA/IOMMU window. With separate PHBs they each have a
separate window. PAPR guests have an always-on guest visible IOMMU.
* Better EEH handling for passthrough devices
EEH is an IBM hardware-assisted mechanism for isolating and safely
resetting devices experiencing hardware faults so they don't bring
down other devices or the system at large. It's roughly similar to
PCIe AER in concept, but has a different IBM specific interface, and
works on both PCI and PCIe devices.
Currently the kernel interfaces for handling EEH events on passthrough
devices will only work if there is a single (host) iommu group in the
vfio container. While lifting that restriction would be nice, it's
quite difficult to do so (it requires keeping state synchronized
between multiple host groups). That also means that an EEH error on
one device could stop another device where that isn't required by the
actual hardware.
The unit of EEH isolation is a PE (Partitionable Endpoint) and
currently there is only one guest PE per vPHB. Changing this might
also be possible, but is again quite complex and may result in
confusing and/or broken distinctions between groups for EEH isolation
and IOMMU isolation purposes.
Placing separate host groups in separate vPHBs sidesteps these
problems.
* Guest NUMA node assignment of devices
PAPR does not (and can't reasonably) use the pxb device. Instead to
allocate devices to different guest NUMA nodes they should be placed
on different vPHBs. Placing them on different PHBs by default allows
NUMA node to be assigned to those PHBs in a straightforward manner.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
5:57 a.m.
On Thu, 5 Jan 2017 16:46:18 +1100
David Gibson <david(a)gibson.dropbear.id.au> wrote:
There was a discussion back in November on the qemu list which
spilled
onto the libvirt list about how to add support for PCIe devices to
POWER VMs, specifically 'pseries' machine type PAPR guests.
Here's a more concrete proposal for how to handle part of this in
future from the libvirt side. Strictly speaking what I'm suggesting
here isn't intrinsically linked to PCIe: it will make adding PCIe
support sanely easier, as well as having a number of advantages for
both PCIe and plain-PCI devices on PAPR guests.
Background:
* Currently the pseries machine type only supports vanilla PCI
buses.
* This is a qemu limitation, not something inherent - PAPR guests
running under PowerVM (the IBM hypervisor) can use passthrough
PCIe devices (PowerVM doesn't emulate devices though).
* In fact the way PCI access is para-virtalized in PAPR makes the
usual distinctions between PCI and PCIe largely disappear
* Presentation of PCIe devices to PAPR guests is unusual
* Unlike x86 - and other "bare metal" platforms, root ports are
not made visible to the guest. i.e. all devices (typically)
appear as though they were integrated devices on x86
* In terms of topology all devices will appear in a way similar to
a vanilla PCI bus, even PCIe devices
* However PCIe extended config space is accessible
* This means libvirt's usual placement of PCIe devices is not
suitable for PAPR guests
* PAPR has its own hotplug mechanism
* This is used instead of standard PCIe hotplug
* This mechanism works for both PCIe and vanilla-PCI devices
* This can hotplug/unplug devices even without a root port P2P
bridge between it and the root "bus
* Multiple independent host bridges are routine on PAPR
* Unlike PC (where all host bridges have multiplexed access to
configuration space) PCI host bridges (PHBs) are truly
independent for PAPR guests (disjoint MMIO regions in system
address space)
* PowerVM typically presents a separate PHB to the guest for each
host slot passed through
The Proposal:
I suggest that libvirt implement a new default algorithm for placing
(i.e. assigning addresses to) both PCI and PCIe devices for (only)
PAPR guests.
The short summary is that by default it should assign each device to a
separate vPHB, creating vPHBs as necessary.
* For passthrough sometimes a group of host devices can't be safely
isolated from each other - this is known as a (host) Partitionable
Endpoint (PE). In this case, if any device in the PE is passed
through to a guest, the whole PE must be passed through to the
same vPHB in the guest. From the guest POV, each vPHB has exactly
one (guest) PE.
* To allow for hotplugged devices, libvirt should also add a number
of additional, empty vPHBs (the PAPR spec allows for hotplug of
PHBs, but this is not yet implemented in qemu). When hotplugging
a new device (or PE) libvirt should locate a vPHB which doesn't
currently contain anything.
* libvirt should only (automatically) add PHBs - never root ports or
other PCI to PCI bridges
In order to handle migration, the vPHBs will need to be represented in
the domain XML, which will also allow the user to override this
topology if they want.
Advantages:
There are still some details I need to figure out w.r.t. handling PCIe
devices (on both the qemu and libvirt sides). However the fact that
One such detail may be that PCIe devices should have the
"ibm,pci-config-space-type" property set to 1 in the DT,
for the driver to be able to access the extended config
space.
PAPR guests don't typically see PCIe root ports means that the
normal
libvirt PCIe allocation scheme won't work. This scheme has several
advantages with or without support for PCIe devices:
* Better performance for 32-bit devices
With multiple devices on a single vPHB they all must share a (fairly
small) 32-bit DMA/IOMMU window. With separate PHBs they each have a
separate window. PAPR guests have an always-on guest visible IOMMU.
* Better EEH handling for passthrough devices
EEH is an IBM hardware-assisted mechanism for isolating and safely
resetting devices experiencing hardware faults so they don't bring
down other devices or the system at large. It's roughly similar to
PCIe AER in concept, but has a different IBM specific interface, and
works on both PCI and PCIe devices.
Currently the kernel interfaces for handling EEH events on passthrough
devices will only work if there is a single (host) iommu group in the
vfio container. While lifting that restriction would be nice, it's
quite difficult to do so (it requires keeping state synchronized
between multiple host groups). That also means that an EEH error on
one device could stop another device where that isn't required by the
actual hardware.
The unit of EEH isolation is a PE (Partitionable Endpoint) and
currently there is only one guest PE per vPHB. Changing this might
also be possible, but is again quite complex and may result in
confusing and/or broken distinctions between groups for EEH isolation
and IOMMU isolation purposes.
Placing separate host groups in separate vPHBs sidesteps these
problems.
* Guest NUMA node assignment of devices
PAPR does not (and can't reasonably) use the pxb device. Instead to
allocate devices to different guest NUMA nodes they should be placed
on different vPHBs. Placing them on different PHBs by default allows
NUMA node to be assigned to those PHBs in a straightforward manner.
5:43 p.m.
On Fri, Jan 06, 2017 at 12:57:58PM +0100, Greg Kurz wrote:
On Thu, 5 Jan 2017 16:46:18 +1100
David Gibson <david(a)gibson.dropbear.id.au> wrote:
> There was a discussion back in November on the qemu list which spilled
> onto the libvirt list about how to add support for PCIe devices to
> POWER VMs, specifically 'pseries' machine type PAPR guests.
>
> Here's a more concrete proposal for how to handle part of this in
> future from the libvirt side. Strictly speaking what I'm suggesting
> here isn't intrinsically linked to PCIe: it will make adding PCIe
> support sanely easier, as well as having a number of advantages for
> both PCIe and plain-PCI devices on PAPR guests.
>
> Background:
>
> * Currently the pseries machine type only supports vanilla PCI
> buses.
> * This is a qemu limitation, not something inherent - PAPR guests
> running under PowerVM (the IBM hypervisor) can use passthrough
> PCIe devices (PowerVM doesn't emulate devices though).
> * In fact the way PCI access is para-virtalized in PAPR makes the
> usual distinctions between PCI and PCIe largely disappear
> * Presentation of PCIe devices to PAPR guests is unusual
> * Unlike x86 - and other "bare metal" platforms, root ports are
> not made visible to the guest. i.e. all devices (typically)
> appear as though they were integrated devices on x86
> * In terms of topology all devices will appear in a way similar to
> a vanilla PCI bus, even PCIe devices
> * However PCIe extended config space is accessible
> * This means libvirt's usual placement of PCIe devices is not
> suitable for PAPR guests
> * PAPR has its own hotplug mechanism
> * This is used instead of standard PCIe hotplug
> * This mechanism works for both PCIe and vanilla-PCI devices
> * This can hotplug/unplug devices even without a root port P2P
> bridge between it and the root "bus
> * Multiple independent host bridges are routine on PAPR
> * Unlike PC (where all host bridges have multiplexed access to
> configuration space) PCI host bridges (PHBs) are truly
> independent for PAPR guests (disjoint MMIO regions in system
> address space)
> * PowerVM typically presents a separate PHB to the guest for each
> host slot passed through
>
> The Proposal:
>
> I suggest that libvirt implement a new default algorithm for placing
> (i.e. assigning addresses to) both PCI and PCIe devices for (only)
> PAPR guests.
>
> The short summary is that by default it should assign each device to a
> separate vPHB, creating vPHBs as necessary.
>
> * For passthrough sometimes a group of host devices can't be safely
> isolated from each other - this is known as a (host) Partitionable
> Endpoint (PE). In this case, if any device in the PE is passed
> through to a guest, the whole PE must be passed through to the
> same vPHB in the guest. From the guest POV, each vPHB has exactly
> one (guest) PE.
> * To allow for hotplugged devices, libvirt should also add a number
> of additional, empty vPHBs (the PAPR spec allows for hotplug of
> PHBs, but this is not yet implemented in qemu). When hotplugging
> a new device (or PE) libvirt should locate a vPHB which doesn't
> currently contain anything.
> * libvirt should only (automatically) add PHBs - never root ports or
> other PCI to PCI bridges
>
> In order to handle migration, the vPHBs will need to be represented in
> the domain XML, which will also allow the user to override this
> topology if they want.
>
> Advantages:
>
> There are still some details I need to figure out w.r.t. handling PCIe
> devices (on both the qemu and libvirt sides). However the fact that
One such detail may be that PCIe devices should have the
"ibm,pci-config-space-type" property set to 1 in the DT,
for the driver to be able to access the extended config
space.
Right.
> PAPR guests don't typically see PCIe root ports means that
the normal
> libvirt PCIe allocation scheme won't work. This scheme has several
> advantages with or without support for PCIe devices:
>
> * Better performance for 32-bit devices
>
> With multiple devices on a single vPHB they all must share a (fairly
> small) 32-bit DMA/IOMMU window. With separate PHBs they each have a
> separate window. PAPR guests have an always-on guest visible IOMMU.
>
> * Better EEH handling for passthrough devices
>
> EEH is an IBM hardware-assisted mechanism for isolating and safely
> resetting devices experiencing hardware faults so they don't bring
> down other devices or the system at large. It's roughly similar to
> PCIe AER in concept, but has a different IBM specific interface, and
> works on both PCI and PCIe devices.
>
> Currently the kernel interfaces for handling EEH events on passthrough
> devices will only work if there is a single (host) iommu group in the
> vfio container. While lifting that restriction would be nice, it's
> quite difficult to do so (it requires keeping state synchronized
> between multiple host groups). That also means that an EEH error on
> one device could stop another device where that isn't required by the
> actual hardware.
>
> The unit of EEH isolation is a PE (Partitionable Endpoint) and
> currently there is only one guest PE per vPHB. Changing this might
> also be possible, but is again quite complex and may result in
> confusing and/or broken distinctions between groups for EEH isolation
> and IOMMU isolation purposes.
>
> Placing separate host groups in separate vPHBs sidesteps these
> problems.
>
> * Guest NUMA node assignment of devices
>
> PAPR does not (and can't reasonably) use the pxb device. Instead to
> allocate devices to different guest NUMA nodes they should be placed
> on different vPHBs. Placing them on different PHBs by default allows
> NUMA node to be assigned to those PHBs in a straightforward manner.
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
9:52 p.m.
On Fri, Jan 06, 2017 at 12:57:58PM +0100, Greg Kurz wrote:
On Thu, 5 Jan 2017 16:46:18 +1100
David Gibson <david(a)gibson.dropbear.id.au> wrote:
> There was a discussion back in November on the qemu list which spilled
> onto the libvirt list about how to add support for PCIe devices to
> POWER VMs, specifically 'pseries' machine type PAPR guests.
>
> Here's a more concrete proposal for how to handle part of this in
> future from the libvirt side. Strictly speaking what I'm suggesting
> here isn't intrinsically linked to PCIe: it will make adding PCIe
> support sanely easier, as well as having a number of advantages for
> both PCIe and plain-PCI devices on PAPR guests.
>
> Background:
>
> * Currently the pseries machine type only supports vanilla PCI
> buses.
> * This is a qemu limitation, not something inherent - PAPR guests
> running under PowerVM (the IBM hypervisor) can use passthrough
> PCIe devices (PowerVM doesn't emulate devices though).
> * In fact the way PCI access is para-virtalized in PAPR makes the
> usual distinctions between PCI and PCIe largely disappear
> * Presentation of PCIe devices to PAPR guests is unusual
> * Unlike x86 - and other "bare metal" platforms, root ports are
> not made visible to the guest. i.e. all devices (typically)
> appear as though they were integrated devices on x86
> * In terms of topology all devices will appear in a way similar to
> a vanilla PCI bus, even PCIe devices
> * However PCIe extended config space is accessible
> * This means libvirt's usual placement of PCIe devices is not
> suitable for PAPR guests
> * PAPR has its own hotplug mechanism
> * This is used instead of standard PCIe hotplug
> * This mechanism works for both PCIe and vanilla-PCI devices
> * This can hotplug/unplug devices even without a root port P2P
> bridge between it and the root "bus
> * Multiple independent host bridges are routine on PAPR
> * Unlike PC (where all host bridges have multiplexed access to
> configuration space) PCI host bridges (PHBs) are truly
> independent for PAPR guests (disjoint MMIO regions in system
> address space)
> * PowerVM typically presents a separate PHB to the guest for each
> host slot passed through
>
> The Proposal:
>
> I suggest that libvirt implement a new default algorithm for placing
> (i.e. assigning addresses to) both PCI and PCIe devices for (only)
> PAPR guests.
>
> The short summary is that by default it should assign each device to a
> separate vPHB, creating vPHBs as necessary.
>
> * For passthrough sometimes a group of host devices can't be safely
> isolated from each other - this is known as a (host) Partitionable
> Endpoint (PE). In this case, if any device in the PE is passed
> through to a guest, the whole PE must be passed through to the
> same vPHB in the guest. From the guest POV, each vPHB has exactly
> one (guest) PE.
> * To allow for hotplugged devices, libvirt should also add a number
> of additional, empty vPHBs (the PAPR spec allows for hotplug of
> PHBs, but this is not yet implemented in qemu). When hotplugging
> a new device (or PE) libvirt should locate a vPHB which doesn't
> currently contain anything.
> * libvirt should only (automatically) add PHBs - never root ports or
> other PCI to PCI bridges
>
> In order to handle migration, the vPHBs will need to be represented in
> the domain XML, which will also allow the user to override this
> topology if they want.
>
> Advantages:
>
> There are still some details I need to figure out w.r.t. handling PCIe
> devices (on both the qemu and libvirt sides). However the fact that
One such detail may be that PCIe devices should have the
"ibm,pci-config-space-type" property set to 1 in the DT,
for the driver to be able to access the extended config
space.
So, we have a bit of an oddity here. It looks like we currently set
'ibm,pci-config-space-type' to 1 in the PHB, rather than individual
device nodes. Which, AFAICT, is simply incorrect in terms of PAPR.
I'm not actually sure if we need to set this dependent on whether the
device actually has extended config space, or if we could just set it
to 1 on every device on a PCIe capable PHB.
> PAPR guests don't typically see PCIe root ports means that
the normal
> libvirt PCIe allocation scheme won't work. This scheme has several
> advantages with or without support for PCIe devices:
>
> * Better performance for 32-bit devices
>
> With multiple devices on a single vPHB they all must share a (fairly
> small) 32-bit DMA/IOMMU window. With separate PHBs they each have a
> separate window. PAPR guests have an always-on guest visible IOMMU.
>
> * Better EEH handling for passthrough devices
>
> EEH is an IBM hardware-assisted mechanism for isolating and safely
> resetting devices experiencing hardware faults so they don't bring
> down other devices or the system at large. It's roughly similar to
> PCIe AER in concept, but has a different IBM specific interface, and
> works on both PCI and PCIe devices.
>
> Currently the kernel interfaces for handling EEH events on passthrough
> devices will only work if there is a single (host) iommu group in the
> vfio container. While lifting that restriction would be nice, it's
> quite difficult to do so (it requires keeping state synchronized
> between multiple host groups). That also means that an EEH error on
> one device could stop another device where that isn't required by the
> actual hardware.
>
> The unit of EEH isolation is a PE (Partitionable Endpoint) and
> currently there is only one guest PE per vPHB. Changing this might
> also be possible, but is again quite complex and may result in
> confusing and/or broken distinctions between groups for EEH isolation
> and IOMMU isolation purposes.
>
> Placing separate host groups in separate vPHBs sidesteps these
> problems.
>
> * Guest NUMA node assignment of devices
>
> PAPR does not (and can't reasonably) use the pxb device. Instead to
> allocate devices to different guest NUMA nodes they should be placed
> on different vPHBs. Placing them on different PHBs by default allows
> NUMA node to be assigned to those PHBs in a straightforward manner.
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
12:19 a.m.
On 12/01/17 14:52, David Gibson wrote:
On Fri, Jan 06, 2017 at 12:57:58PM +0100, Greg Kurz wrote:
> On Thu, 5 Jan 2017 16:46:18 +1100
> David Gibson <david(a)gibson.dropbear.id.au> wrote:
>
>> There was a discussion back in November on the qemu list which spilled
>> onto the libvirt list about how to add support for PCIe devices to
>> POWER VMs, specifically 'pseries' machine type PAPR guests.
>>
>> Here's a more concrete proposal for how to handle part of this in
>> future from the libvirt side. Strictly speaking what I'm suggesting
>> here isn't intrinsically linked to PCIe: it will make adding PCIe
>> support sanely easier, as well as having a number of advantages for
>> both PCIe and plain-PCI devices on PAPR guests.
>>
>> Background:
>>
>> * Currently the pseries machine type only supports vanilla PCI
>> buses.
>> * This is a qemu limitation, not something inherent - PAPR guests
>> running under PowerVM (the IBM hypervisor) can use passthrough
>> PCIe devices (PowerVM doesn't emulate devices though).
>> * In fact the way PCI access is para-virtalized in PAPR makes the
>> usual distinctions between PCI and PCIe largely disappear
>> * Presentation of PCIe devices to PAPR guests is unusual
>> * Unlike x86 - and other "bare metal" platforms, root ports are
>> not made visible to the guest. i.e. all devices (typically)
>> appear as though they were integrated devices on x86
>> * In terms of topology all devices will appear in a way similar to
>> a vanilla PCI bus, even PCIe devices
>> * However PCIe extended config space is accessible
>> * This means libvirt's usual placement of PCIe devices is not
>> suitable for PAPR guests
>> * PAPR has its own hotplug mechanism
>> * This is used instead of standard PCIe hotplug
>> * This mechanism works for both PCIe and vanilla-PCI devices
>> * This can hotplug/unplug devices even without a root port P2P
>> bridge between it and the root "bus
>> * Multiple independent host bridges are routine on PAPR
>> * Unlike PC (where all host bridges have multiplexed access to
>> configuration space) PCI host bridges (PHBs) are truly
>> independent for PAPR guests (disjoint MMIO regions in system
>> address space)
>> * PowerVM typically presents a separate PHB to the guest for each
>> host slot passed through
>>
>> The Proposal:
>>
>> I suggest that libvirt implement a new default algorithm for placing
>> (i.e. assigning addresses to) both PCI and PCIe devices for (only)
>> PAPR guests.
>>
>> The short summary is that by default it should assign each device to a
>> separate vPHB, creating vPHBs as necessary.
>>
>> * For passthrough sometimes a group of host devices can't be safely
>> isolated from each other - this is known as a (host) Partitionable
>> Endpoint (PE). In this case, if any device in the PE is passed
>> through to a guest, the whole PE must be passed through to the
>> same vPHB in the guest. From the guest POV, each vPHB has exactly
>> one (guest) PE.
>> * To allow for hotplugged devices, libvirt should also add a number
>> of additional, empty vPHBs (the PAPR spec allows for hotplug of
>> PHBs, but this is not yet implemented in qemu). When hotplugging
>> a new device (or PE) libvirt should locate a vPHB which doesn't
>> currently contain anything.
>> * libvirt should only (automatically) add PHBs - never root ports or
>> other PCI to PCI bridges
>>
>> In order to handle migration, the vPHBs will need to be represented in
>> the domain XML, which will also allow the user to override this
>> topology if they want.
>>
>> Advantages:
>>
>> There are still some details I need to figure out w.r.t. handling PCIe
>> devices (on both the qemu and libvirt sides). However the fact that
>
> One such detail may be that PCIe devices should have the
> "ibm,pci-config-space-type" property set to 1 in the DT,
> for the driver to be able to access the extended config
> space.
So, we have a bit of an oddity here. It looks like we currently set
'ibm,pci-config-space-type' to 1 in the PHB, rather than individual
device nodes. Which, AFAICT, is simply incorrect in terms of PAPR.
I asked Paul how to read the spec and this is rather correct but not enough
- having type=1 on a PHB means that extended access requests can go behind
it but underlying devices and bridges still need to have type=1 if they
support extended space. Having type set to 0 (or none at all) on a PHB
would mean that extended config space is not available on anything under
this PHB.
--
Alexey
3:09 a.m.
On Thu, 12 Jan 2017 17:19:40 +1100
Alexey Kardashevskiy <aik(a)ozlabs.ru> wrote:
On 12/01/17 14:52, David Gibson wrote:
> On Fri, Jan 06, 2017 at 12:57:58PM +0100, Greg Kurz wrote:
>> On Thu, 5 Jan 2017 16:46:18 +1100
>> David Gibson <david(a)gibson.dropbear.id.au> wrote:
>>
>>> There was a discussion back in November on the qemu list which spilled
>>> onto the libvirt list about how to add support for PCIe devices to
>>> POWER VMs, specifically 'pseries' machine type PAPR guests.
>>>
>>> Here's a more concrete proposal for how to handle part of this in
>>> future from the libvirt side. Strictly speaking what I'm suggesting
>>> here isn't intrinsically linked to PCIe: it will make adding PCIe
>>> support sanely easier, as well as having a number of advantages for
>>> both PCIe and plain-PCI devices on PAPR guests.
>>>
>>> Background:
>>>
>>> * Currently the pseries machine type only supports vanilla PCI
>>> buses.
>>> * This is a qemu limitation, not something inherent - PAPR guests
>>> running under PowerVM (the IBM hypervisor) can use passthrough
>>> PCIe devices (PowerVM doesn't emulate devices though).
>>> * In fact the way PCI access is para-virtalized in PAPR makes the
>>> usual distinctions between PCI and PCIe largely disappear
>>> * Presentation of PCIe devices to PAPR guests is unusual
>>> * Unlike x86 - and other "bare metal" platforms, root ports
are
>>> not made visible to the guest. i.e. all devices (typically)
>>> appear as though they were integrated devices on x86
>>> * In terms of topology all devices will appear in a way similar to
>>> a vanilla PCI bus, even PCIe devices
>>> * However PCIe extended config space is accessible
>>> * This means libvirt's usual placement of PCIe devices is not
>>> suitable for PAPR guests
>>> * PAPR has its own hotplug mechanism
>>> * This is used instead of standard PCIe hotplug
>>> * This mechanism works for both PCIe and vanilla-PCI devices
>>> * This can hotplug/unplug devices even without a root port P2P
>>> bridge between it and the root "bus
>>> * Multiple independent host bridges are routine on PAPR
>>> * Unlike PC (where all host bridges have multiplexed access to
>>> configuration space) PCI host bridges (PHBs) are truly
>>> independent for PAPR guests (disjoint MMIO regions in system
>>> address space)
>>> * PowerVM typically presents a separate PHB to the guest for each
>>> host slot passed through
>>>
>>> The Proposal:
>>>
>>> I suggest that libvirt implement a new default algorithm for placing
>>> (i.e. assigning addresses to) both PCI and PCIe devices for (only)
>>> PAPR guests.
>>>
>>> The short summary is that by default it should assign each device to a
>>> separate vPHB, creating vPHBs as necessary.
>>>
>>> * For passthrough sometimes a group of host devices can't be safely
>>> isolated from each other - this is known as a (host) Partitionable
>>> Endpoint (PE). In this case, if any device in the PE is passed
>>> through to a guest, the whole PE must be passed through to the
>>> same vPHB in the guest. From the guest POV, each vPHB has exactly
>>> one (guest) PE.
>>> * To allow for hotplugged devices, libvirt should also add a number
>>> of additional, empty vPHBs (the PAPR spec allows for hotplug of
>>> PHBs, but this is not yet implemented in qemu). When hotplugging
>>> a new device (or PE) libvirt should locate a vPHB which doesn't
>>> currently contain anything.
>>> * libvirt should only (automatically) add PHBs - never root ports or
>>> other PCI to PCI bridges
>>>
>>> In order to handle migration, the vPHBs will need to be represented in
>>> the domain XML, which will also allow the user to override this
>>> topology if they want.
>>>
>>> Advantages:
>>>
>>> There are still some details I need to figure out w.r.t. handling PCIe
>>> devices (on both the qemu and libvirt sides). However the fact that
>>
>> One such detail may be that PCIe devices should have the
>> "ibm,pci-config-space-type" property set to 1 in the DT,
>> for the driver to be able to access the extended config
>> space.
>
> So, we have a bit of an oddity here. It looks like we currently set
> 'ibm,pci-config-space-type' to 1 in the PHB, rather than individual
> device nodes. Which, AFAICT, is simply incorrect in terms of PAPR.
I asked Paul how to read the spec and this is rather correct but not enough
- having type=1 on a PHB means that extended access requests can go behind
it but underlying devices and bridges still need to have type=1 if they
support extended space. Having type set to 0 (or none at all) on a PHB
would mean that extended config space is not available on anything under
this PHB.
I have the very same understanding of the spec (LoPAPR March 2016):
R1–9.1.8–2. All IOAs that implement PCI-X Mode 2 or PCI Express must supply the
“ibm,pci-con-
fig-space-type” property (see Section B.6.5.1.1.1‚ “Properties for Children of PCI Host
Bridges‚” on
page 703).
Implementation Note: The “ibm,pci-config-space-type” property in Requirement R1–9.1.8–2 is
added for
platforms that support I/O fabric and IOAs that implement PCI-X Mode 2, and PCI Express.
To access the
extended configuration space provided by PCI-X Mode 2 and PCI Express, all I/O fabric
leading up to an IOA
must support a 12-bit register number. In other words, if a platform implementation has a
conventional PCI bridge
leading up to an IOA that implements PCI-X Mode 2, the platform will not be able to
provide access to the
extended configuration space of that IOA. The “ibm,config-space-type” property in the
IOA's OF node
is used by device drivers to determine if an IOA’s extended configuration space can be
accessed.
and
B.6.5.1.1.1 Properties for Children of PCI Host Bridges
“ibm,pci-config-space-type”
property name: Indicates if the platform supports access to an extended configuration
address space from the PHB
up to and including this node.
0 = Platform supports only an eight bit register number for configuration address space
accesses.
1 = Platform supports a twelve bit register number for configuration address space
accesses.
This property may be provided in all PHB nodes and their children.
Note: The absence of this property implies the platform supports only an eight bit
register number for configura-
tion address space accesses.
And incidentally, this is what the linux kernel currently expects. See these lines
from arch/powerpc/kernel/pci_dn.c:
struct pci_dn *pci_add_device_node_info(struct pci_controller *hose,
struct device_node *dn)
{
const __be32 *type = of_get_property(dn, "ibm,pci-config-space-type",
NULL);
.
.
.
/* Extended config space */
pdn->pci_ext_config_space = (type && of_read_number(type, 1) == 1);
I had to rework Alexey's "spapr_pci: Create PCI-express root bus by
default"
patch to be able to see the extended config space of a vfio-pci device:
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -1052,6 +1052,9 @@ static int spapr_populate_pci_child_dt(PCIDevice *dev, void *fdt,
int offset,
_FDT(fdt_setprop(fdt, offset, "reg", (uint8_t *)rp.reg, rp.reg_len));
_FDT(fdt_setprop(fdt, offset, "assigned-addresses",
(uint8_t *)rp.assigned, rp.assigned_len));
+ if (sphb->pcie_root && pci_is_express(dev)) {
+ _FDT(fdt_setprop_cell(fdt, offset, "ibm,pci-config-space-type", 0x1));
+ }
return 0;
}
10:48 p.m.
On Thu, Jan 12, 2017 at 10:09:03AM +0100, Greg Kurz wrote:
On Thu, 12 Jan 2017 17:19:40 +1100
Alexey Kardashevskiy <aik(a)ozlabs.ru> wrote:
> On 12/01/17 14:52, David Gibson wrote:
> > On Fri, Jan 06, 2017 at 12:57:58PM +0100, Greg Kurz wrote:
> >> On Thu, 5 Jan 2017 16:46:18 +1100
> >> David Gibson <david(a)gibson.dropbear.id.au> wrote:
> >>
> >>> There was a discussion back in November on the qemu list which spilled
> >>> onto the libvirt list about how to add support for PCIe devices to
> >>> POWER VMs, specifically 'pseries' machine type PAPR guests.
> >>>
> >>> Here's a more concrete proposal for how to handle part of this in
> >>> future from the libvirt side. Strictly speaking what I'm
suggesting
> >>> here isn't intrinsically linked to PCIe: it will make adding PCIe
> >>> support sanely easier, as well as having a number of advantages for
> >>> both PCIe and plain-PCI devices on PAPR guests.
> >>>
> >>> Background:
> >>>
> >>> * Currently the pseries machine type only supports vanilla PCI
> >>> buses.
> >>> * This is a qemu limitation, not something inherent - PAPR guests
> >>> running under PowerVM (the IBM hypervisor) can use passthrough
> >>> PCIe devices (PowerVM doesn't emulate devices though).
> >>> * In fact the way PCI access is para-virtalized in PAPR makes the
> >>> usual distinctions between PCI and PCIe largely disappear
> >>> * Presentation of PCIe devices to PAPR guests is unusual
> >>> * Unlike x86 - and other "bare metal" platforms, root
ports are
> >>> not made visible to the guest. i.e. all devices (typically)
> >>> appear as though they were integrated devices on x86
> >>> * In terms of topology all devices will appear in a way similar to
> >>> a vanilla PCI bus, even PCIe devices
> >>> * However PCIe extended config space is accessible
> >>> * This means libvirt's usual placement of PCIe devices is not
> >>> suitable for PAPR guests
> >>> * PAPR has its own hotplug mechanism
> >>> * This is used instead of standard PCIe hotplug
> >>> * This mechanism works for both PCIe and vanilla-PCI devices
> >>> * This can hotplug/unplug devices even without a root port P2P
> >>> bridge between it and the root "bus
> >>> * Multiple independent host bridges are routine on PAPR
> >>> * Unlike PC (where all host bridges have multiplexed access to
> >>> configuration space) PCI host bridges (PHBs) are truly
> >>> independent for PAPR guests (disjoint MMIO regions in system
> >>> address space)
> >>> * PowerVM typically presents a separate PHB to the guest for each
> >>> host slot passed through
> >>>
> >>> The Proposal:
> >>>
> >>> I suggest that libvirt implement a new default algorithm for placing
> >>> (i.e. assigning addresses to) both PCI and PCIe devices for (only)
> >>> PAPR guests.
> >>>
> >>> The short summary is that by default it should assign each device to a
> >>> separate vPHB, creating vPHBs as necessary.
> >>>
> >>> * For passthrough sometimes a group of host devices can't be
safely
> >>> isolated from each other - this is known as a (host) Partitionable
> >>> Endpoint (PE). In this case, if any device in the PE is passed
> >>> through to a guest, the whole PE must be passed through to the
> >>> same vPHB in the guest. From the guest POV, each vPHB has exactly
> >>> one (guest) PE.
> >>> * To allow for hotplugged devices, libvirt should also add a number
> >>> of additional, empty vPHBs (the PAPR spec allows for hotplug of
> >>> PHBs, but this is not yet implemented in qemu). When hotplugging
> >>> a new device (or PE) libvirt should locate a vPHB which
doesn't
> >>> currently contain anything.
> >>> * libvirt should only (automatically) add PHBs - never root ports or
> >>> other PCI to PCI bridges
> >>>
> >>> In order to handle migration, the vPHBs will need to be represented in
> >>> the domain XML, which will also allow the user to override this
> >>> topology if they want.
> >>>
> >>> Advantages:
> >>>
> >>> There are still some details I need to figure out w.r.t. handling PCIe
> >>> devices (on both the qemu and libvirt sides). However the fact that
> >>
> >> One such detail may be that PCIe devices should have the
> >> "ibm,pci-config-space-type" property set to 1 in the DT,
> >> for the driver to be able to access the extended config
> >> space.
> >
> > So, we have a bit of an oddity here. It looks like we currently set
> > 'ibm,pci-config-space-type' to 1 in the PHB, rather than individual
> > device nodes. Which, AFAICT, is simply incorrect in terms of PAPR.
>
>
> I asked Paul how to read the spec and this is rather correct but not enough
> - having type=1 on a PHB means that extended access requests can go behind
> it but underlying devices and bridges still need to have type=1 if they
> support extended space. Having type set to 0 (or none at all) on a PHB
> would mean that extended config space is not available on anything under
> this PHB.
>
I have the very same understanding of the spec (LoPAPR March 2016):
R1–9.1.8–2. All IOAs that implement PCI-X Mode 2 or PCI Express must supply the
“ibm,pci-con-
fig-space-type” property (see Section B.6.5.1.1.1‚ “Properties for Children of PCI Host
Bridges‚” on
page 703).
Implementation Note: The “ibm,pci-config-space-type” property in Requirement R1–9.1.8–2
is added for
platforms that support I/O fabric and IOAs that implement PCI-X Mode 2, and PCI Express.
To access the
extended configuration space provided by PCI-X Mode 2 and PCI Express, all I/O fabric
leading up to an IOA
must support a 12-bit register number. In other words, if a platform implementation has a
conventional PCI bridge
leading up to an IOA that implements PCI-X Mode 2, the platform will not be able to
provide access to the
extended configuration space of that IOA. The “ibm,config-space-type” property in the
IOA's OF node
is used by device drivers to determine if an IOA’s extended configuration space can be
accessed.
and
B.6.5.1.1.1 Properties for Children of PCI Host Bridges
“ibm,pci-config-space-type”
property name: Indicates if the platform supports access to an extended configuration
address space from the PHB
up to and including this node.
0 = Platform supports only an eight bit register number for configuration address space
accesses.
1 = Platform supports a twelve bit register number for configuration address space
accesses.
This property may be provided in all PHB nodes and their children.
Note: The absence of this property implies the platform supports only an eight bit
register number for configura-
tion address space accesses.
And incidentally, this is what the linux kernel currently expects. See these lines
from arch/powerpc/kernel/pci_dn.c:
struct pci_dn *pci_add_device_node_info(struct pci_controller *hose,
struct device_node *dn)
{
const __be32 *type = of_get_property(dn, "ibm,pci-config-space-type",
NULL);
.
.
.
/* Extended config space */
pdn->pci_ext_config_space = (type && of_read_number(type, 1) == 1);
Ok, thanks for the information.
I had to rework Alexey's "spapr_pci: Create PCI-express root
bus by default"
patch to be able to see the extended config space of a vfio-pci device:
Ah! Is there an easy command line way to verify that extended config
space is accessible?
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -1052,6 +1052,9 @@ static int spapr_populate_pci_child_dt(PCIDevice *dev, void *fdt,
int offset,
_FDT(fdt_setprop(fdt, offset, "reg", (uint8_t *)rp.reg, rp.reg_len));
_FDT(fdt_setprop(fdt, offset, "assigned-addresses",
(uint8_t *)rp.assigned, rp.assigned_len));
+ if (sphb->pcie_root && pci_is_express(dev)) {
+ _FDT(fdt_setprop_cell(fdt, offset, "ibm,pci-config-space-type",
0x1));
+ }
return 0;
}
I'm looking at merging the patch below into ppc-for-2.9 shortly. It
basically combines a revised version of the pcie option part of
Alexey's patch with your DT patch above. For now it only allows PCIe
with an explicit option, changing the default waits on more
investigation of how to not break things with libvirt.
From e14702a0bb76900077d82510280e1a023c384b72 Mon Sep 17 00:00:00 2001
From: David Gibson <david(a)gibson.dropbear.id.au>
Date: Fri, 13 Jan 2017 15:45:30 +1100
Subject: [PATCH] spapr_pci: Allow PCI-Express devices
Currently the PCI host bridge (PHB) for the pseries machine type always
creates a plain PCI bus (TYPE_PCI_BUS). This means that qemu won't allow
PCIe devices to be attached, and consequently, PCIe devices can't be used
on pseries.
This limitation isn't inherent in the PAPR specification. In fact, because
access to the PCI bus is paravirtualized via hypercalls, from the guest
point of view there is very little difference between a plain PCI and PCIe
bus under PAPR.
So, to allow PCIe devices, add a "pcie" option to the PHB device, which
when enabled creates a PCIe bus. In addition, on PCIe devices under such
a PHB, we publish the device tree flag which indicates that the guest may
access PCIe extended config space (via RTAS calls which we've already
implemented).
Note that this doesn't change the default PHB to PCIe. We probably want to
do that in future, but we need to sort out some possible breakages with
libvirt and existing users first. This at least lets knowledgeable users
experiment with PCIe devices in the meantime.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
hw/ppc/spapr_pci.c | 8 +++++++-
include/hw/pci-host/spapr.h | 1 +
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index fd6fc1d..d17df1f 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -1053,6 +1053,10 @@ static int spapr_populate_pci_child_dt(PCIDevice *dev, void *fdt,
int offset,
_FDT(fdt_setprop(fdt, offset, "assigned-addresses",
(uint8_t *)rp.assigned, rp.assigned_len));
+ if (sphb->pcie && pci_is_express(dev)) {
+ _FDT(fdt_setprop_cell(fdt, offset, "ibm,pci-config-space-type", 0x1));
+ }
+
return 0;
}
@@ -1434,7 +1438,8 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp)
bus = pci_register_bus(dev, NULL,
pci_spapr_set_irq, pci_spapr_map_irq, sphb,
&sphb->memspace, &sphb->iospace,
- PCI_DEVFN(0, 0), PCI_NUM_PINS, TYPE_PCI_BUS);
+ PCI_DEVFN(0, 0), PCI_NUM_PINS,
+ sphb->pcie ? TYPE_PCIE_BUS : TYPE_PCI_BUS);
phb->bus = bus;
qbus_set_hotplug_handler(BUS(phb->bus), DEVICE(sphb), NULL);
@@ -1592,6 +1597,7 @@ static Property spapr_phb_properties[] = {
DEFINE_PROP_UINT32("numa_node", sPAPRPHBState, numa_node, -1),
DEFINE_PROP_BOOL("pre-2.8-migration", sPAPRPHBState,
pre_2_8_migration, false),
+ DEFINE_PROP_BOOL("pcie", sPAPRPHBState, pcie, false),
DEFINE_PROP_END_OF_LIST(),
};
diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h
index 092294e..deed2cc 100644
--- a/include/hw/pci-host/spapr.h
+++ b/include/hw/pci-host/spapr.h
@@ -79,6 +79,7 @@ struct sPAPRPHBState {
uint64_t dma64_win_addr;
uint32_t numa_node;
+ bool pcie;
/* Fields for migration compatibility hacks */
bool pre_2_8_migration;
--
2.9.3
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
12:03 a.m.
On 13/01/17 15:48, David Gibson wrote:
On Thu, Jan 12, 2017 at 10:09:03AM +0100, Greg Kurz wrote:
> On Thu, 12 Jan 2017 17:19:40 +1100
> Alexey Kardashevskiy <aik(a)ozlabs.ru> wrote:
>
>> On 12/01/17 14:52, David Gibson wrote:
>>> On Fri, Jan 06, 2017 at 12:57:58PM +0100, Greg Kurz wrote:
>>>> On Thu, 5 Jan 2017 16:46:18 +1100
>>>> David Gibson <david(a)gibson.dropbear.id.au> wrote:
>>>>
>>>>> There was a discussion back in November on the qemu list which
spilled
>>>>> onto the libvirt list about how to add support for PCIe devices to
>>>>> POWER VMs, specifically 'pseries' machine type PAPR guests.
>>>>>
>>>>> Here's a more concrete proposal for how to handle part of this
in
>>>>> future from the libvirt side. Strictly speaking what I'm
suggesting
>>>>> here isn't intrinsically linked to PCIe: it will make adding
PCIe
>>>>> support sanely easier, as well as having a number of advantages for
>>>>> both PCIe and plain-PCI devices on PAPR guests.
>>>>>
>>>>> Background:
>>>>>
>>>>> * Currently the pseries machine type only supports vanilla PCI
>>>>> buses.
>>>>> * This is a qemu limitation, not something inherent - PAPR
guests
>>>>> running under PowerVM (the IBM hypervisor) can use passthrough
>>>>> PCIe devices (PowerVM doesn't emulate devices though).
>>>>> * In fact the way PCI access is para-virtalized in PAPR makes
the
>>>>> usual distinctions between PCI and PCIe largely disappear
>>>>> * Presentation of PCIe devices to PAPR guests is unusual
>>>>> * Unlike x86 - and other "bare metal" platforms, root
ports are
>>>>> not made visible to the guest. i.e. all devices (typically)
>>>>> appear as though they were integrated devices on x86
>>>>> * In terms of topology all devices will appear in a way similar
to
>>>>> a vanilla PCI bus, even PCIe devices
>>>>> * However PCIe extended config space is accessible
>>>>> * This means libvirt's usual placement of PCIe devices is
not
>>>>> suitable for PAPR guests
>>>>> * PAPR has its own hotplug mechanism
>>>>> * This is used instead of standard PCIe hotplug
>>>>> * This mechanism works for both PCIe and vanilla-PCI devices
>>>>> * This can hotplug/unplug devices even without a root port P2P
>>>>> bridge between it and the root "bus
>>>>> * Multiple independent host bridges are routine on PAPR
>>>>> * Unlike PC (where all host bridges have multiplexed access to
>>>>> configuration space) PCI host bridges (PHBs) are truly
>>>>> independent for PAPR guests (disjoint MMIO regions in system
>>>>> address space)
>>>>> * PowerVM typically presents a separate PHB to the guest for
each
>>>>> host slot passed through
>>>>>
>>>>> The Proposal:
>>>>>
>>>>> I suggest that libvirt implement a new default algorithm for placing
>>>>> (i.e. assigning addresses to) both PCI and PCIe devices for (only)
>>>>> PAPR guests.
>>>>>
>>>>> The short summary is that by default it should assign each device to
a
>>>>> separate vPHB, creating vPHBs as necessary.
>>>>>
>>>>> * For passthrough sometimes a group of host devices can't be
safely
>>>>> isolated from each other - this is known as a (host)
Partitionable
>>>>> Endpoint (PE). In this case, if any device in the PE is passed
>>>>> through to a guest, the whole PE must be passed through to the
>>>>> same vPHB in the guest. From the guest POV, each vPHB has
exactly
>>>>> one (guest) PE.
>>>>> * To allow for hotplugged devices, libvirt should also add a
number
>>>>> of additional, empty vPHBs (the PAPR spec allows for hotplug of
>>>>> PHBs, but this is not yet implemented in qemu). When
hotplugging
>>>>> a new device (or PE) libvirt should locate a vPHB which
doesn't
>>>>> currently contain anything.
>>>>> * libvirt should only (automatically) add PHBs - never root ports
or
>>>>> other PCI to PCI bridges
>>>>>
>>>>> In order to handle migration, the vPHBs will need to be represented
in
>>>>> the domain XML, which will also allow the user to override this
>>>>> topology if they want.
>>>>>
>>>>> Advantages:
>>>>>
>>>>> There are still some details I need to figure out w.r.t. handling
PCIe
>>>>> devices (on both the qemu and libvirt sides). However the fact that
>>>>
>>>> One such detail may be that PCIe devices should have the
>>>> "ibm,pci-config-space-type" property set to 1 in the DT,
>>>> for the driver to be able to access the extended config
>>>> space.
>>>
>>> So, we have a bit of an oddity here. It looks like we currently set
>>> 'ibm,pci-config-space-type' to 1 in the PHB, rather than individual
>>> device nodes. Which, AFAICT, is simply incorrect in terms of PAPR.
>>
>>
>> I asked Paul how to read the spec and this is rather correct but not enough
>> - having type=1 on a PHB means that extended access requests can go behind
>> it but underlying devices and bridges still need to have type=1 if they
>> support extended space. Having type set to 0 (or none at all) on a PHB
>> would mean that extended config space is not available on anything under
>> this PHB.
>>
>
> I have the very same understanding of the spec (LoPAPR March 2016):
>
> R1–9.1.8–2. All IOAs that implement PCI-X Mode 2 or PCI Express must supply the
“ibm,pci-con-
> fig-space-type” property (see Section B.6.5.1.1.1‚ “Properties for Children of PCI
Host Bridges‚” on
> page 703).
>
> Implementation Note: The “ibm,pci-config-space-type” property in Requirement
R1–9.1.8–2 is added for
> platforms that support I/O fabric and IOAs that implement PCI-X Mode 2, and PCI
Express. To access the
> extended configuration space provided by PCI-X Mode 2 and PCI Express, all I/O fabric
leading up to an IOA
> must support a 12-bit register number. In other words, if a platform implementation
has a conventional PCI bridge
> leading up to an IOA that implements PCI-X Mode 2, the platform will not be able to
provide access to the
> extended configuration space of that IOA. The “ibm,config-space-type” property in the
IOA's OF node
> is used by device drivers to determine if an IOA’s extended configuration space can
be accessed.
>
> and
>
> B.6.5.1.1.1 Properties for Children of PCI Host Bridges
>
> “ibm,pci-config-space-type”
> property name: Indicates if the platform supports access to an extended configuration
address space from the PHB
> up to and including this node.
> 0 = Platform supports only an eight bit register number for configuration address
space accesses.
> 1 = Platform supports a twelve bit register number for configuration address space
accesses.
> This property may be provided in all PHB nodes and their children.
> Note: The absence of this property implies the platform supports only an eight bit
register number for configura-
> tion address space accesses.
>
>
> And incidentally, this is what the linux kernel currently expects. See these lines
> from arch/powerpc/kernel/pci_dn.c:
>
> struct pci_dn *pci_add_device_node_info(struct pci_controller *hose,
> struct device_node *dn)
> {
> const __be32 *type = of_get_property(dn,
"ibm,pci-config-space-type", NULL);
> .
> .
> .
> /* Extended config space */
> pdn->pci_ext_config_space = (type && of_read_number(type, 1) ==
1);
Ok, thanks for the information.
> I had to rework Alexey's "spapr_pci: Create PCI-express root bus by
default"
> patch to be able to see the extended config space of a vfio-pci device:
Ah! Is there an easy command line way to verify that extended config
space is accessible?
I do "lspci -vvs "0003:01:00.3 and look for "Capabilities: [xxx v1]"
where
xxx >= 0x100.
--
Alexey
2:58 a.m.
On Fri, 13 Jan 2017 15:48:31 +1100
David Gibson <david(a)gibson.dropbear.id.au> wrote:
On Thu, Jan 12, 2017 at 10:09:03AM +0100, Greg Kurz wrote:
> On Thu, 12 Jan 2017 17:19:40 +1100
> Alexey Kardashevskiy <aik(a)ozlabs.ru> wrote:
>
> > On 12/01/17 14:52, David Gibson wrote:
> > > On Fri, Jan 06, 2017 at 12:57:58PM +0100, Greg Kurz wrote:
> > >> On Thu, 5 Jan 2017 16:46:18 +1100
> > >> David Gibson <david(a)gibson.dropbear.id.au> wrote:
> > >>
> > >>> There was a discussion back in November on the qemu list which
spilled
> > >>> onto the libvirt list about how to add support for PCIe devices
to
> > >>> POWER VMs, specifically 'pseries' machine type PAPR
guests.
> > >>>
> > >>> Here's a more concrete proposal for how to handle part of this
in
> > >>> future from the libvirt side. Strictly speaking what I'm
suggesting
> > >>> here isn't intrinsically linked to PCIe: it will make adding
PCIe
> > >>> support sanely easier, as well as having a number of advantages
for
> > >>> both PCIe and plain-PCI devices on PAPR guests.
> > >>>
> > >>> Background:
> > >>>
> > >>> * Currently the pseries machine type only supports vanilla PCI
> > >>> buses.
> > >>> * This is a qemu limitation, not something inherent - PAPR
guests
> > >>> running under PowerVM (the IBM hypervisor) can use
passthrough
> > >>> PCIe devices (PowerVM doesn't emulate devices though).
> > >>> * In fact the way PCI access is para-virtalized in PAPR makes
the
> > >>> usual distinctions between PCI and PCIe largely disappear
> > >>> * Presentation of PCIe devices to PAPR guests is unusual
> > >>> * Unlike x86 - and other "bare metal" platforms,
root ports are
> > >>> not made visible to the guest. i.e. all devices (typically)
> > >>> appear as though they were integrated devices on x86
> > >>> * In terms of topology all devices will appear in a way
similar to
> > >>> a vanilla PCI bus, even PCIe devices
> > >>> * However PCIe extended config space is accessible
> > >>> * This means libvirt's usual placement of PCIe devices is
not
> > >>> suitable for PAPR guests
> > >>> * PAPR has its own hotplug mechanism
> > >>> * This is used instead of standard PCIe hotplug
> > >>> * This mechanism works for both PCIe and vanilla-PCI devices
> > >>> * This can hotplug/unplug devices even without a root port
P2P
> > >>> bridge between it and the root "bus
> > >>> * Multiple independent host bridges are routine on PAPR
> > >>> * Unlike PC (where all host bridges have multiplexed access
to
> > >>> configuration space) PCI host bridges (PHBs) are truly
> > >>> independent for PAPR guests (disjoint MMIO regions in
system
> > >>> address space)
> > >>> * PowerVM typically presents a separate PHB to the guest for
each
> > >>> host slot passed through
> > >>>
> > >>> The Proposal:
> > >>>
> > >>> I suggest that libvirt implement a new default algorithm for
placing
> > >>> (i.e. assigning addresses to) both PCI and PCIe devices for
(only)
> > >>> PAPR guests.
> > >>>
> > >>> The short summary is that by default it should assign each device
to a
> > >>> separate vPHB, creating vPHBs as necessary.
> > >>>
> > >>> * For passthrough sometimes a group of host devices can't be
safely
> > >>> isolated from each other - this is known as a (host)
Partitionable
> > >>> Endpoint (PE). In this case, if any device in the PE is
passed
> > >>> through to a guest, the whole PE must be passed through to
the
> > >>> same vPHB in the guest. From the guest POV, each vPHB has
exactly
> > >>> one (guest) PE.
> > >>> * To allow for hotplugged devices, libvirt should also add a
number
> > >>> of additional, empty vPHBs (the PAPR spec allows for hotplug
of
> > >>> PHBs, but this is not yet implemented in qemu). When
hotplugging
> > >>> a new device (or PE) libvirt should locate a vPHB which
doesn't
> > >>> currently contain anything.
> > >>> * libvirt should only (automatically) add PHBs - never root
ports or
> > >>> other PCI to PCI bridges
> > >>>
> > >>> In order to handle migration, the vPHBs will need to be
represented in
> > >>> the domain XML, which will also allow the user to override this
> > >>> topology if they want.
> > >>>
> > >>> Advantages:
> > >>>
> > >>> There are still some details I need to figure out w.r.t. handling
PCIe
> > >>> devices (on both the qemu and libvirt sides). However the fact
that
> > >>
> > >> One such detail may be that PCIe devices should have the
> > >> "ibm,pci-config-space-type" property set to 1 in the DT,
> > >> for the driver to be able to access the extended config
> > >> space.
> > >
> > > So, we have a bit of an oddity here. It looks like we currently set
> > > 'ibm,pci-config-space-type' to 1 in the PHB, rather than
individual
> > > device nodes. Which, AFAICT, is simply incorrect in terms of PAPR.
> >
> >
> > I asked Paul how to read the spec and this is rather correct but not enough
> > - having type=1 on a PHB means that extended access requests can go behind
> > it but underlying devices and bridges still need to have type=1 if they
> > support extended space. Having type set to 0 (or none at all) on a PHB
> > would mean that extended config space is not available on anything under
> > this PHB.
> >
>
> I have the very same understanding of the spec (LoPAPR March 2016):
>
> R1–9.1.8–2. All IOAs that implement PCI-X Mode 2 or PCI Express must supply the
“ibm,pci-con-
> fig-space-type” property (see Section B.6.5.1.1.1‚ “Properties for Children of PCI
Host Bridges‚” on
> page 703).
>
> Implementation Note: The “ibm,pci-config-space-type” property in Requirement
R1–9.1.8–2 is added for
> platforms that support I/O fabric and IOAs that implement PCI-X Mode 2, and PCI
Express. To access the
> extended configuration space provided by PCI-X Mode 2 and PCI Express, all I/O
fabric leading up to an IOA
> must support a 12-bit register number. In other words, if a platform implementation
has a conventional PCI bridge
> leading up to an IOA that implements PCI-X Mode 2, the platform will not be able to
provide access to the
> extended configuration space of that IOA. The “ibm,config-space-type” property in
the IOA's OF node
> is used by device drivers to determine if an IOA’s extended configuration space can
be accessed.
>
> and
>
> B.6.5.1.1.1 Properties for Children of PCI Host Bridges
>
> “ibm,pci-config-space-type”
> property name: Indicates if the platform supports access to an extended
configuration address space from the PHB
> up to and including this node.
> 0 = Platform supports only an eight bit register number for configuration address
space accesses.
> 1 = Platform supports a twelve bit register number for configuration address space
accesses.
> This property may be provided in all PHB nodes and their children.
> Note: The absence of this property implies the platform supports only an eight bit
register number for configura-
> tion address space accesses.
>
>
> And incidentally, this is what the linux kernel currently expects. See these lines
> from arch/powerpc/kernel/pci_dn.c:
>
> struct pci_dn *pci_add_device_node_info(struct pci_controller *hose,
> struct device_node *dn)
> {
> const __be32 *type = of_get_property(dn,
"ibm,pci-config-space-type", NULL);
> .
> .
> .
> /* Extended config space */
> pdn->pci_ext_config_space = (type && of_read_number(type, 1) ==
1);
Ok, thanks for the information.
> I had to rework Alexey's "spapr_pci: Create PCI-express root bus by
default"
> patch to be able to see the extended config space of a vfio-pci device:
Ah! Is there an easy command line way to verify that extended config
space is accessible?
'lspci -vv' prints the capabilities from the config space with their
hex offset between [ ]. Extended config space starts at offset 0x100.
> --- a/hw/ppc/spapr_pci.c
> +++ b/hw/ppc/spapr_pci.c
> @@ -1052,6 +1052,9 @@ static int spapr_populate_pci_child_dt(PCIDevice *dev, void
*fdt, int offset,
> _FDT(fdt_setprop(fdt, offset, "reg", (uint8_t *)rp.reg,
rp.reg_len));
> _FDT(fdt_setprop(fdt, offset, "assigned-addresses",
> (uint8_t *)rp.assigned, rp.assigned_len));
> + if (sphb->pcie_root && pci_is_express(dev)) {
> + _FDT(fdt_setprop_cell(fdt, offset, "ibm,pci-config-space-type",
0x1));
> + }
>
> return 0;
> }
I'm looking at merging the patch below into ppc-for-2.9 shortly. It
basically combines a revised version of the pcie option part of
Alexey's patch with your DT patch above. For now it only allows PCIe
with an explicit option, changing the default waits on more
investigation of how to not break things with libvirt.
From e14702a0bb76900077d82510280e1a023c384b72 Mon Sep 17 00:00:00 2001
From: David Gibson <david(a)gibson.dropbear.id.au>
Date: Fri, 13 Jan 2017 15:45:30 +1100
Subject: [PATCH] spapr_pci: Allow PCI-Express devices
Currently the PCI host bridge (PHB) for the pseries machine type always
creates a plain PCI bus (TYPE_PCI_BUS). This means that qemu won't allow
PCIe devices to be attached, and consequently, PCIe devices can't be used
on pseries.
Well, PCIe devices can still be attached actually (at least I could attach
a vfio-pci device linked to a PCIe device on the host). But, they appear
as legacy PCI devices in the guest, without the extended config space.
This limitation isn't inherent in the PAPR specification. In
fact, because
access to the PCI bus is paravirtualized via hypercalls, from the guest
point of view there is very little difference between a plain PCI and PCIe
bus under PAPR.
So, to allow PCIe devices, add a "pcie" option to the PHB device, which
when enabled creates a PCIe bus. In addition, on PCIe devices under such
a PHB, we publish the device tree flag which indicates that the guest may
access PCIe extended config space (via RTAS calls which we've already
implemented).
Note that this doesn't change the default PHB to PCIe. We probably want to
do that in future, but we need to sort out some possible breakages with
libvirt and existing users first. This at least lets knowledgeable users
experiment with PCIe devices in the meantime.
Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
FWIW,
Reviewed-by: Greg Kurz <groug(a)kaod.org>
---
hw/ppc/spapr_pci.c | 8 +++++++-
include/hw/pci-host/spapr.h | 1 +
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index fd6fc1d..d17df1f 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -1053,6 +1053,10 @@ static int spapr_populate_pci_child_dt(PCIDevice *dev, void *fdt,
int offset,
_FDT(fdt_setprop(fdt, offset, "assigned-addresses",
(uint8_t *)rp.assigned, rp.assigned_len));
+ if (sphb->pcie && pci_is_express(dev)) {
+ _FDT(fdt_setprop_cell(fdt, offset, "ibm,pci-config-space-type",
0x1));
+ }
+
return 0;
}
@@ -1434,7 +1438,8 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp)
bus = pci_register_bus(dev, NULL,
pci_spapr_set_irq, pci_spapr_map_irq, sphb,
&sphb->memspace, &sphb->iospace,
- PCI_DEVFN(0, 0), PCI_NUM_PINS, TYPE_PCI_BUS);
+ PCI_DEVFN(0, 0), PCI_NUM_PINS,
+ sphb->pcie ? TYPE_PCIE_BUS : TYPE_PCI_BUS);
phb->bus = bus;
qbus_set_hotplug_handler(BUS(phb->bus), DEVICE(sphb), NULL);
@@ -1592,6 +1597,7 @@ static Property spapr_phb_properties[] = {
DEFINE_PROP_UINT32("numa_node", sPAPRPHBState, numa_node, -1),
DEFINE_PROP_BOOL("pre-2.8-migration", sPAPRPHBState,
pre_2_8_migration, false),
+ DEFINE_PROP_BOOL("pcie", sPAPRPHBState, pcie, false),
DEFINE_PROP_END_OF_LIST(),
};
diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h
index 092294e..deed2cc 100644
--- a/include/hw/pci-host/spapr.h
+++ b/include/hw/pci-host/spapr.h
@@ -79,6 +79,7 @@ struct sPAPRPHBState {
uint64_t dma64_win_addr;
uint32_t numa_node;
+ bool pcie;
/* Fields for migration compatibility hacks */
bool pre_2_8_migration;
5:08 a.m.
New subject: [libvirt] [Qemu-ppc] Proposal PCI/PCIe device placement on PAPR guests
David,
I don't see the "spapr_pci: Allow PCI-Express devices" patch in your
ppc-for-2.9 tree. Do you still consider merging it ?
Cheers.
--
Greg
On Fri, 13 Jan 2017 09:58:28 +0100
Greg Kurz <groug(a)kaod.org> wrote:
On Fri, 13 Jan 2017 15:48:31 +1100
David Gibson <david(a)gibson.dropbear.id.au> wrote:
> On Thu, Jan 12, 2017 at 10:09:03AM +0100, Greg Kurz wrote:
> > On Thu, 12 Jan 2017 17:19:40 +1100
> > Alexey Kardashevskiy <aik(a)ozlabs.ru> wrote:
> >
> > > On 12/01/17 14:52, David Gibson wrote:
> > > > On Fri, Jan 06, 2017 at 12:57:58PM +0100, Greg Kurz wrote:
> > > >> On Thu, 5 Jan 2017 16:46:18 +1100
> > > >> David Gibson <david(a)gibson.dropbear.id.au> wrote:
> > > >>
> > > >>> There was a discussion back in November on the qemu list
which spilled
> > > >>> onto the libvirt list about how to add support for PCIe
devices to
> > > >>> POWER VMs, specifically 'pseries' machine type PAPR
guests.
> > > >>>
> > > >>> Here's a more concrete proposal for how to handle part of
this in
> > > >>> future from the libvirt side. Strictly speaking what I'm
suggesting
> > > >>> here isn't intrinsically linked to PCIe: it will make
adding PCIe
> > > >>> support sanely easier, as well as having a number of
advantages for
> > > >>> both PCIe and plain-PCI devices on PAPR guests.
> > > >>>
> > > >>> Background:
> > > >>>
> > > >>> * Currently the pseries machine type only supports vanilla
PCI
> > > >>> buses.
> > > >>> * This is a qemu limitation, not something inherent -
PAPR guests
> > > >>> running under PowerVM (the IBM hypervisor) can use
passthrough
> > > >>> PCIe devices (PowerVM doesn't emulate devices
though).
> > > >>> * In fact the way PCI access is para-virtalized in PAPR
makes the
> > > >>> usual distinctions between PCI and PCIe largely
disappear
> > > >>> * Presentation of PCIe devices to PAPR guests is unusual
> > > >>> * Unlike x86 - and other "bare metal"
platforms, root ports are
> > > >>> not made visible to the guest. i.e. all devices
(typically)
> > > >>> appear as though they were integrated devices on x86
> > > >>> * In terms of topology all devices will appear in a way
similar to
> > > >>> a vanilla PCI bus, even PCIe devices
> > > >>> * However PCIe extended config space is accessible
> > > >>> * This means libvirt's usual placement of PCIe
devices is not
> > > >>> suitable for PAPR guests
> > > >>> * PAPR has its own hotplug mechanism
> > > >>> * This is used instead of standard PCIe hotplug
> > > >>> * This mechanism works for both PCIe and vanilla-PCI
devices
> > > >>> * This can hotplug/unplug devices even without a root
port P2P
> > > >>> bridge between it and the root "bus
> > > >>> * Multiple independent host bridges are routine on PAPR
> > > >>> * Unlike PC (where all host bridges have multiplexed
access to
> > > >>> configuration space) PCI host bridges (PHBs) are truly
> > > >>> independent for PAPR guests (disjoint MMIO regions in
system
> > > >>> address space)
> > > >>> * PowerVM typically presents a separate PHB to the guest
for each
> > > >>> host slot passed through
> > > >>>
> > > >>> The Proposal:
> > > >>>
> > > >>> I suggest that libvirt implement a new default algorithm for
placing
> > > >>> (i.e. assigning addresses to) both PCI and PCIe devices for
(only)
> > > >>> PAPR guests.
> > > >>>
> > > >>> The short summary is that by default it should assign each
device to a
> > > >>> separate vPHB, creating vPHBs as necessary.
> > > >>>
> > > >>> * For passthrough sometimes a group of host devices
can't be safely
> > > >>> isolated from each other - this is known as a (host)
Partitionable
> > > >>> Endpoint (PE). In this case, if any device in the PE is
passed
> > > >>> through to a guest, the whole PE must be passed through
to the
> > > >>> same vPHB in the guest. From the guest POV, each vPHB
has exactly
> > > >>> one (guest) PE.
> > > >>> * To allow for hotplugged devices, libvirt should also add
a number
> > > >>> of additional, empty vPHBs (the PAPR spec allows for
hotplug of
> > > >>> PHBs, but this is not yet implemented in qemu). When
hotplugging
> > > >>> a new device (or PE) libvirt should locate a vPHB which
doesn't
> > > >>> currently contain anything.
> > > >>> * libvirt should only (automatically) add PHBs - never root
ports or
> > > >>> other PCI to PCI bridges
> > > >>>
> > > >>> In order to handle migration, the vPHBs will need to be
represented in
> > > >>> the domain XML, which will also allow the user to override
this
> > > >>> topology if they want.
> > > >>>
> > > >>> Advantages:
> > > >>>
> > > >>> There are still some details I need to figure out w.r.t.
handling PCIe
> > > >>> devices (on both the qemu and libvirt sides). However the
fact that
> > > >>
> > > >> One such detail may be that PCIe devices should have the
> > > >> "ibm,pci-config-space-type" property set to 1 in the
DT,
> > > >> for the driver to be able to access the extended config
> > > >> space.
> > > >
> > > > So, we have a bit of an oddity here. It looks like we currently set
> > > > 'ibm,pci-config-space-type' to 1 in the PHB, rather than
individual
> > > > device nodes. Which, AFAICT, is simply incorrect in terms of PAPR.
> > >
> > >
> > > I asked Paul how to read the spec and this is rather correct but not
enough
> > > - having type=1 on a PHB means that extended access requests can go
behind
> > > it but underlying devices and bridges still need to have type=1 if they
> > > support extended space. Having type set to 0 (or none at all) on a PHB
> > > would mean that extended config space is not available on anything under
> > > this PHB.
> > >
> >
> > I have the very same understanding of the spec (LoPAPR March 2016):
> >
> > R1–9.1.8–2. All IOAs that implement PCI-X Mode 2 or PCI Express must supply the
“ibm,pci-con-
> > fig-space-type” property (see Section B.6.5.1.1.1‚ “Properties for Children of
PCI Host Bridges‚” on
> > page 703).
> >
> > Implementation Note: The “ibm,pci-config-space-type” property in Requirement
R1–9.1.8–2 is added for
> > platforms that support I/O fabric and IOAs that implement PCI-X Mode 2, and PCI
Express. To access the
> > extended configuration space provided by PCI-X Mode 2 and PCI Express, all I/O
fabric leading up to an IOA
> > must support a 12-bit register number. In other words, if a platform
implementation has a conventional PCI bridge
> > leading up to an IOA that implements PCI-X Mode 2, the platform will not be
able to provide access to the
> > extended configuration space of that IOA. The “ibm,config-space-type” property
in the IOA's OF node
> > is used by device drivers to determine if an IOA’s extended configuration space
can be accessed.
> >
> > and
> >
> > B.6.5.1.1.1 Properties for Children of PCI Host Bridges
> >
> > “ibm,pci-config-space-type”
> > property name: Indicates if the platform supports access to an extended
configuration address space from the PHB
> > up to and including this node.
> > 0 = Platform supports only an eight bit register number for configuration
address space accesses.
> > 1 = Platform supports a twelve bit register number for configuration address
space accesses.
> > This property may be provided in all PHB nodes and their children.
> > Note: The absence of this property implies the platform supports only an eight
bit register number for configura-
> > tion address space accesses.
> >
> >
> > And incidentally, this is what the linux kernel currently expects. See these
lines
> > from arch/powerpc/kernel/pci_dn.c:
> >
> > struct pci_dn *pci_add_device_node_info(struct pci_controller *hose,
> > struct device_node *dn)
> > {
> > const __be32 *type = of_get_property(dn,
"ibm,pci-config-space-type", NULL);
> > .
> > .
> > .
> > /* Extended config space */
> > pdn->pci_ext_config_space = (type && of_read_number(type, 1)
== 1);
>
> Ok, thanks for the information.
>
> > I had to rework Alexey's "spapr_pci: Create PCI-express root bus by
default"
> > patch to be able to see the extended config space of a vfio-pci device:
>
> Ah! Is there an easy command line way to verify that extended config
> space is accessible?
>
'lspci -vv' prints the capabilities from the config space with their
hex offset between [ ]. Extended config space starts at offset 0x100.
> > --- a/hw/ppc/spapr_pci.c
> > +++ b/hw/ppc/spapr_pci.c
> > @@ -1052,6 +1052,9 @@ static int spapr_populate_pci_child_dt(PCIDevice *dev,
void *fdt, int offset,
> > _FDT(fdt_setprop(fdt, offset, "reg", (uint8_t *)rp.reg,
rp.reg_len));
> > _FDT(fdt_setprop(fdt, offset, "assigned-addresses",
> > (uint8_t *)rp.assigned, rp.assigned_len));
> > + if (sphb->pcie_root && pci_is_express(dev)) {
> > + _FDT(fdt_setprop_cell(fdt, offset,
"ibm,pci-config-space-type", 0x1));
> > + }
> >
> > return 0;
> > }
>
> I'm looking at merging the patch below into ppc-for-2.9 shortly. It
> basically combines a revised version of the pcie option part of
> Alexey's patch with your DT patch above. For now it only allows PCIe
> with an explicit option, changing the default waits on more
> investigation of how to not break things with libvirt.
>
> From e14702a0bb76900077d82510280e1a023c384b72 Mon Sep 17 00:00:00 2001
> From: David Gibson <david(a)gibson.dropbear.id.au>
> Date: Fri, 13 Jan 2017 15:45:30 +1100
> Subject: [PATCH] spapr_pci: Allow PCI-Express devices
>
> Currently the PCI host bridge (PHB) for the pseries machine type always
> creates a plain PCI bus (TYPE_PCI_BUS). This means that qemu won't allow
> PCIe devices to be attached, and consequently, PCIe devices can't be used
> on pseries.
>
Well, PCIe devices can still be attached actually (at least I could attach
a vfio-pci device linked to a PCIe device on the host). But, they appear
as legacy PCI devices in the guest, without the extended config space.
> This limitation isn't inherent in the PAPR specification. In fact, because
> access to the PCI bus is paravirtualized via hypercalls, from the guest
> point of view there is very little difference between a plain PCI and PCIe
> bus under PAPR.
>
> So, to allow PCIe devices, add a "pcie" option to the PHB device, which
> when enabled creates a PCIe bus. In addition, on PCIe devices under such
> a PHB, we publish the device tree flag which indicates that the guest may
> access PCIe extended config space (via RTAS calls which we've already
> implemented).
>
> Note that this doesn't change the default PHB to PCIe. We probably want to
> do that in future, but we need to sort out some possible breakages with
> libvirt and existing users first. This at least lets knowledgeable users
> experiment with PCIe devices in the meantime.
>
> Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
FWIW,
Reviewed-by: Greg Kurz <groug(a)kaod.org>
> ---
> hw/ppc/spapr_pci.c | 8 +++++++-
> include/hw/pci-host/spapr.h | 1 +
> 2 files changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> index fd6fc1d..d17df1f 100644
> --- a/hw/ppc/spapr_pci.c
> +++ b/hw/ppc/spapr_pci.c
> @@ -1053,6 +1053,10 @@ static int spapr_populate_pci_child_dt(PCIDevice *dev, void
*fdt, int offset,
> _FDT(fdt_setprop(fdt, offset, "assigned-addresses",
> (uint8_t *)rp.assigned, rp.assigned_len));
>
> + if (sphb->pcie && pci_is_express(dev)) {
> + _FDT(fdt_setprop_cell(fdt, offset, "ibm,pci-config-space-type",
0x1));
> + }
> +
> return 0;
> }
>
> @@ -1434,7 +1438,8 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp)
> bus = pci_register_bus(dev, NULL,
> pci_spapr_set_irq, pci_spapr_map_irq, sphb,
> &sphb->memspace, &sphb->iospace,
> - PCI_DEVFN(0, 0), PCI_NUM_PINS, TYPE_PCI_BUS);
> + PCI_DEVFN(0, 0), PCI_NUM_PINS,
> + sphb->pcie ? TYPE_PCIE_BUS : TYPE_PCI_BUS);
> phb->bus = bus;
> qbus_set_hotplug_handler(BUS(phb->bus), DEVICE(sphb), NULL);
>
> @@ -1592,6 +1597,7 @@ static Property spapr_phb_properties[] = {
> DEFINE_PROP_UINT32("numa_node", sPAPRPHBState, numa_node, -1),
> DEFINE_PROP_BOOL("pre-2.8-migration", sPAPRPHBState,
> pre_2_8_migration, false),
> + DEFINE_PROP_BOOL("pcie", sPAPRPHBState, pcie, false),
> DEFINE_PROP_END_OF_LIST(),
> };
>
> diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h
> index 092294e..deed2cc 100644
> --- a/include/hw/pci-host/spapr.h
> +++ b/include/hw/pci-host/spapr.h
> @@ -79,6 +79,7 @@ struct sPAPRPHBState {
> uint64_t dma64_win_addr;
>
> uint32_t numa_node;
> + bool pcie;
>
> /* Fields for migration compatibility hacks */
> bool pre_2_8_migration;
8:11 p.m.
New subject: [libvirt] [Qemu-ppc] Proposal PCI/PCIe device placement on PAPR guests
On Wed, Feb 22, 2017 at 12:08:25PM +0100, Greg Kurz wrote:
David,
I don't see the "spapr_pci: Allow PCI-Express devices" patch in your
ppc-for-2.9 tree. Do you still consider merging it ?
No. After discussions with Marcel Apfelbaum and others I'm looking at
a different approach for PCIe support.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
1:23 a.m.
New subject: [libvirt] [Qemu-ppc] Proposal PCI/PCIe device placement on PAPR guests
On Thu, 23 Feb 2017 13:11:52 +1100
David Gibson <david(a)gibson.dropbear.id.au> wrote:
On Wed, Feb 22, 2017 at 12:08:25PM +0100, Greg Kurz wrote:
> David,
>
> I don't see the "spapr_pci: Allow PCI-Express devices" patch in your
> ppc-for-2.9 tree. Do you still consider merging it ?
No. After discussions with Marcel Apfelbaum and others I'm looking at
a different approach for PCIe support.
Oh, I had missed that... Is it the following thread ?
https://lists.gnu.org/archive/html/qemu-ppc/2017-02/msg00203.html
4:57 p.m.
New subject: [libvirt] [Qemu-ppc] Proposal PCI/PCIe device placement on PAPR guests
On Thu, Feb 23, 2017 at 08:23:50AM +0100, Greg Kurz wrote:
On Thu, 23 Feb 2017 13:11:52 +1100
David Gibson <david(a)gibson.dropbear.id.au> wrote:
> On Wed, Feb 22, 2017 at 12:08:25PM +0100, Greg Kurz wrote:
> > David,
> >
> > I don't see the "spapr_pci: Allow PCI-Express devices" patch in
your
> > ppc-for-2.9 tree. Do you still consider merging it ?
>
> No. After discussions with Marcel Apfelbaum and others I'm looking at
> a different approach for PCIe support.
>
Oh, I had missed that... Is it the following thread ?
https://lists.gnu.org/archive/html/qemu-ppc/2017-02/msg00203.html
That's the one.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
6:01 a.m.
Resending because of bad qemu-devel address...
On Thu, 5 Jan 2017 16:46:18 +1100
David Gibson <david(a)gibson.dropbear.id.au> wrote:
There was a discussion back in November on the qemu list which
spilled
onto the libvirt list about how to add support for PCIe devices to
POWER VMs, specifically 'pseries' machine type PAPR guests.
Here's a more concrete proposal for how to handle part of this in
future from the libvirt side. Strictly speaking what I'm suggesting
here isn't intrinsically linked to PCIe: it will make adding PCIe
support sanely easier, as well as having a number of advantages for
both PCIe and plain-PCI devices on PAPR guests.
Background:
* Currently the pseries machine type only supports vanilla PCI
buses.
* This is a qemu limitation, not something inherent - PAPR guests
running under PowerVM (the IBM hypervisor) can use passthrough
PCIe devices (PowerVM doesn't emulate devices though).
* In fact the way PCI access is para-virtalized in PAPR makes the
usual distinctions between PCI and PCIe largely disappear
* Presentation of PCIe devices to PAPR guests is unusual
* Unlike x86 - and other "bare metal" platforms, root ports are
not made visible to the guest. i.e. all devices (typically)
appear as though they were integrated devices on x86
* In terms of topology all devices will appear in a way similar to
a vanilla PCI bus, even PCIe devices
* However PCIe extended config space is accessible
* This means libvirt's usual placement of PCIe devices is not
suitable for PAPR guests
* PAPR has its own hotplug mechanism
* This is used instead of standard PCIe hotplug
* This mechanism works for both PCIe and vanilla-PCI devices
* This can hotplug/unplug devices even without a root port P2P
bridge between it and the root "bus
* Multiple independent host bridges are routine on PAPR
* Unlike PC (where all host bridges have multiplexed access to
configuration space) PCI host bridges (PHBs) are truly
independent for PAPR guests (disjoint MMIO regions in system
address space)
* PowerVM typically presents a separate PHB to the guest for each
host slot passed through
The Proposal:
I suggest that libvirt implement a new default algorithm for placing
(i.e. assigning addresses to) both PCI and PCIe devices for (only)
PAPR guests.
The short summary is that by default it should assign each device to a
separate vPHB, creating vPHBs as necessary.
* For passthrough sometimes a group of host devices can't be safely
isolated from each other - this is known as a (host) Partitionable
Endpoint (PE). In this case, if any device in the PE is passed
through to a guest, the whole PE must be passed through to the
same vPHB in the guest. From the guest POV, each vPHB has exactly
one (guest) PE.
* To allow for hotplugged devices, libvirt should also add a number
of additional, empty vPHBs (the PAPR spec allows for hotplug of
PHBs, but this is not yet implemented in qemu). When hotplugging
a new device (or PE) libvirt should locate a vPHB which doesn't
currently contain anything.
* libvirt should only (automatically) add PHBs - never root ports or
other PCI to PCI bridges
In order to handle migration, the vPHBs will need to be represented in
the domain XML, which will also allow the user to override this
topology if they want.
Advantages:
There are still some details I need to figure out w.r.t. handling PCIe
devices (on both the qemu and libvirt sides). However the fact that
One such detail may be that PCIe devices should have the
"ibm,pci-config-space-type" property set to 1 in the DT,
for the driver to be able to access the extended config
space.
PAPR guests don't typically see PCIe root ports means that the
normal
libvirt PCIe allocation scheme won't work. This scheme has several
advantages with or without support for PCIe devices:
* Better performance for 32-bit devices
With multiple devices on a single vPHB they all must share a (fairly
small) 32-bit DMA/IOMMU window. With separate PHBs they each have a
separate window. PAPR guests have an always-on guest visible IOMMU.
* Better EEH handling for passthrough devices
EEH is an IBM hardware-assisted mechanism for isolating and safely
resetting devices experiencing hardware faults so they don't bring
down other devices or the system at large. It's roughly similar to
PCIe AER in concept, but has a different IBM specific interface, and
works on both PCI and PCIe devices.
Currently the kernel interfaces for handling EEH events on passthrough
devices will only work if there is a single (host) iommu group in the
vfio container. While lifting that restriction would be nice, it's
quite difficult to do so (it requires keeping state synchronized
between multiple host groups). That also means that an EEH error on
one device could stop another device where that isn't required by the
actual hardware.
The unit of EEH isolation is a PE (Partitionable Endpoint) and
currently there is only one guest PE per vPHB. Changing this might
also be possible, but is again quite complex and may result in
confusing and/or broken distinctions between groups for EEH isolation
and IOMMU isolation purposes.
Placing separate host groups in separate vPHBs sidesteps these
problems.
* Guest NUMA node assignment of devices
PAPR does not (and can't reasonably) use the pxb device. Instead to
allocate devices to different guest NUMA nodes they should be placed
on different vPHBs. Placing them on different PHBs by default allows
NUMA node to be assigned to those PHBs in a straightforward manner.
11:34 a.m.
[Added Laine to CC, fixed qemu-devel address]
On Thu, 2017-01-05 at 16:46 +1100, David Gibson wrote:
[...]
* To allow for hotplugged devices, libvirt should also add a
number
of additional, empty vPHBs (the PAPR spec allows for hotplug of
PHBs, but this is not yet implemented in qemu).
"A number" here will have to mean "one", same number of
empty PCIe Root Ports libvirt will add to a newly-defined
q35 guest.
When hotplugging
a new device (or PE) libvirt should locate a vPHB which doesn't
currently contain anything.
This will need to be a PHB-specific behavior, because at the
moment libvirt will happily pick one of the empty slots in
an existing PHB.
* libvirt should only (automatically) add PHBs - never root ports
or
other PCI to PCI bridges
In order to handle migration, the vPHBs will need to be represented in
the domain XML, which will also allow the user to override this
topology if they want.
We'll have to decide how to represent them in the XML, but
that's basically your average bikeshedding.
Overall, the plan seems entirely reasonable to me.
It's pretty clear at this point that pseries guest are
different enough in their handling of PCI that none of
the address allocation algorithms currently implemented
in libvirt could be quite adapted to work with it, so
a custom one is in order.
--
Andrea Bolognani / Red Hat / Virtualization
5:46 p.m.
On Fri, Jan 06, 2017 at 06:34:29PM +0100, Andrea Bolognani wrote:
[Added Laine to CC, fixed qemu-devel address]
On Thu, 2017-01-05 at 16:46 +1100, David Gibson wrote:
[...]
> * To allow for hotplugged devices, libvirt should also add a number
> of additional, empty vPHBs (the PAPR spec allows for hotplug of
> PHBs, but this is not yet implemented in qemu).
"A number" here will have to mean "one", same number of
empty PCIe Root Ports libvirt will add to a newly-defined
q35 guest.
Umm.. why?
> When hotplugging
> a new device (or PE) libvirt should locate a vPHB which doesn't
> currently contain anything.
This will need to be a PHB-specific behavior, because at the
moment libvirt will happily pick one of the empty slots in
an existing PHB.
Exactly. Well, whether it's PHB model specific or machine type
specific is up to you really. We can only have PAPR PHBs on a PAPR
machine type, so it's kind of arbitrary.
> * libvirt should only (automatically) add PHBs - never root ports or
> other PCI to PCI bridges
>
> In order to handle migration, the vPHBs will need to be represented in
> the domain XML, which will also allow the user to override this
> topology if they want.
We'll have to decide how to represent them in the XML, but
that's basically your average bikeshedding.
Right. Maybe we'd best get started with it, in the hopes of finishing
it in the forseeable future.
Overall, the plan seems entirely reasonable to me.
It's pretty clear at this point that pseries guest are
different enough in their handling of PCI that none of
the address allocation algorithms currently implemented
in libvirt could be quite adapted to work with it, so
a custom one is in order.
Yes, that was my conclusion as well.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
4:31 a.m.
On Mon, 2017-01-09 at 10:46 +1100, David Gibson wrote:
> > * To allow for hotplugged devices, libvirt should also
add a number
> > of additional, empty vPHBs (the PAPR spec allows for hotplug of
> > PHBs, but this is not yet implemented in qemu).
>
> "A number" here will have to mean "one", same number of
> empty PCIe Root Ports libvirt will add to a newly-defined
> q35 guest.
Umm.. why?
Because some applications using libvirt would inevitably
start relying on the fact that such spare PHBs are
available, locking us into providing at least the same
number forever. In other words, increasing the amount at
a later time is always possible, but decreasing it isn't.
We did the same when we started automatically adding PCIe
Root Ports to q35 machines.
The rationale is that having a single spare hotpluggable
slot is extremely convenient for basic usage, eg. a simple
guest created by someone who's not necessarily very
familiar with virtualization; on the other hand, if you
are actually deploying in production you ought to conduct
proper capacity planning and figure out in advance how
many devices you're likely to need to hotplug throughout
the guest's life.
Of course this all will be moot once we can hotplug PHBs :)
--
Andrea Bolognani / Red Hat / Virtualization
8:52 a.m.
On 01/12/2017 05:31 AM, Andrea Bolognani wrote:
On Mon, 2017-01-09 at 10:46 +1100, David Gibson wrote:
>>> * To allow for hotplugged devices, libvirt should also add a number
>>> of additional, empty vPHBs (the PAPR spec allows for hotplug of
>>> PHBs, but this is not yet implemented in qemu).
>>
>> "A number" here will have to mean "one", same number of
>> empty PCIe Root Ports libvirt will add to a newly-defined
>> q35 guest.
>
> Umm.. why?
Because some applications using libvirt would inevitably
start relying on the fact that such spare PHBs are
available, locking us into providing at least the same
number forever. In other words, increasing the amount at
a later time is always possible, but decreasing it isn't.
We did the same when we started automatically adding PCIe
Root Ports to q35 machines.
The rationale is that having a single spare hotpluggable
slot is extremely convenient for basic usage, eg. a simple
guest created by someone who's not necessarily very
familiar with virtualization; on the other hand, if you
are actually deploying in production you ought to conduct
proper capacity planning and figure out in advance how
many devices you're likely to need to hotplug throughout
the guest's life.
And of course the reason we don't want to add "too many" extra
controllers by default is so that we don't end up with *all* guests
burdened with extra hardware they don't need or want. The libguestfs
appliance is one example of a libvirt consumer that definitely doesn't
want extra baggage in its guests - guest startup time is very important
to libguestfs, so any addition to the hardware list is looked upon with
disappointment.
Of course this all will be moot once we can hotplug PHBs :)
Will the guest OSes handle that properly? I remember being told that
Linux, for example, doesn't scan the new bus for devices when a new
controller is added, making it pointless to hotplug a PCI controller (as
usual, it could be that I'm remembering incorrectly...)
10:35 a.m.
Quoting Laine Stump (2017-01-12 08:52:10)
On 01/12/2017 05:31 AM, Andrea Bolognani wrote:
> On Mon, 2017-01-09 at 10:46 +1100, David Gibson wrote:
>>>> * To allow for hotplugged devices, libvirt should also add a number
>>>> of additional, empty vPHBs (the PAPR spec allows for hotplug of
>>>> PHBs, but this is not yet implemented in qemu).
>>>
>>> "A number" here will have to mean "one", same number of
>>> empty PCIe Root Ports libvirt will add to a newly-defined
>>> q35 guest.
>>
>> Umm.. why?
>
> Because some applications using libvirt would inevitably
> start relying on the fact that such spare PHBs are
> available, locking us into providing at least the same
> number forever. In other words, increasing the amount at
> a later time is always possible, but decreasing it isn't.
> We did the same when we started automatically adding PCIe
> Root Ports to q35 machines.
>
> The rationale is that having a single spare hotpluggable
> slot is extremely convenient for basic usage, eg. a simple
> guest created by someone who's not necessarily very
> familiar with virtualization; on the other hand, if you
> are actually deploying in production you ought to conduct
> proper capacity planning and figure out in advance how
> many devices you're likely to need to hotplug throughout
> the guest's life.
And of course the reason we don't want to add "too many" extra
controllers by default is so that we don't end up with *all* guests
burdened with extra hardware they don't need or want. The libguestfs
appliance is one example of a libvirt consumer that definitely doesn't
want extra baggage in its guests - guest startup time is very important
to libguestfs, so any addition to the hardware list is looked upon with
disappointment.
>
> Of course this all will be moot once we can hotplug PHBs :)
Will the guest OSes handle that properly? I remember being told that
I believe on pseries we *do* scan for devices on the PHB as part of
bringing the PHB online in the hotplug path. But I'm not sure that
matters (see below).
Linux, for example, doesn't scan the new bus for devices when a
new
controller is added, making it pointless to hotplug a PCI controller (as
usual, it could be that I'm remembering incorrectly...)
Wouldn't that only be an issue if we hotplugged a PHB that already had
PCI devices on the bus? That only seems possible if we had a way to
signal phb hotplug *after* we've hotplugged some PCI devices on the bus,
which means we'd need some interface to trigger hotplug beyond the
standard 'device_add' calls, e.g.:
device_add spapr-pci-host-bridge,hotplug-deferred=true,id=phb2,index=2
device_add virtio-net-pci,bus=phb2.0,...,hotplug-deferred=true
device_signal_hotplug phb2
That's actually akin to how it's normally done on pHyp (not only for PHB
hotplug, but for PCI hotplug in general, which is why this could be
reasonably expected to work on pseries guests), but it seems quite a bit
different from how we'd normally handle this on kvm, which I think would
be something more like:
device_add spapr-pci-host-bridge,id=phb2,index=2
<wait for hotplug completion event>
device_add virtio-net-pci,bus=phb2.0,...
In which case it doesn't really matter if the guest scans the bus at
hotplug time or not. Is there some other scenario where this might
arise?
11:53 a.m.
On 01/12/2017 11:35 AM, Michael Roth wrote:
Quoting Laine Stump (2017-01-12 08:52:10)
> On 01/12/2017 05:31 AM, Andrea Bolognani wrote:
>> On Mon, 2017-01-09 at 10:46 +1100, David Gibson wrote:
>>>>> * To allow for hotplugged devices, libvirt should also add a
number
>>>>> of additional, empty vPHBs (the PAPR spec allows for hotplug
of
>>>>> PHBs, but this is not yet implemented in qemu).
>>>>
>>>> "A number" here will have to mean "one", same number
of
>>>> empty PCIe Root Ports libvirt will add to a newly-defined
>>>> q35 guest.
>>>
>>> Umm.. why?
>>
>> Because some applications using libvirt would inevitably
>> start relying on the fact that such spare PHBs are
>> available, locking us into providing at least the same
>> number forever. In other words, increasing the amount at
>> a later time is always possible, but decreasing it isn't.
>> We did the same when we started automatically adding PCIe
>> Root Ports to q35 machines.
>>
>> The rationale is that having a single spare hotpluggable
>> slot is extremely convenient for basic usage, eg. a simple
>> guest created by someone who's not necessarily very
>> familiar with virtualization; on the other hand, if you
>> are actually deploying in production you ought to conduct
>> proper capacity planning and figure out in advance how
>> many devices you're likely to need to hotplug throughout
>> the guest's life.
>
> And of course the reason we don't want to add "too many" extra
> controllers by default is so that we don't end up with *all* guests
> burdened with extra hardware they don't need or want. The libguestfs
> appliance is one example of a libvirt consumer that definitely doesn't
> want extra baggage in its guests - guest startup time is very important
> to libguestfs, so any addition to the hardware list is looked upon with
> disappointment.
>
>>
>> Of course this all will be moot once we can hotplug PHBs :)
>
> Will the guest OSes handle that properly? I remember being told that
I believe on pseries we *do* scan for devices on the PHB as part of
bringing the PHB online in the hotplug path. But I'm not sure that
matters (see below).
> Linux, for example, doesn't scan the new bus for devices when a new
> controller is added, making it pointless to hotplug a PCI controller (as
> usual, it could be that I'm remembering incorrectly...)
>
Wouldn't that only be an issue if we hotplugged a PHB that already had
PCI devices on the bus?
Yeah you're right, I'm probably remembering the wrong problem and wrong
reason for the problem. I just remember there was *some* issue about
hotplugging new PCI controllers. Possibly the internal representation of
the bus hierarchy wasn't updated unless you forced a rescan of all the
devices or something? My memory of it is vague, I just remember being
told it wasn't just a case of the controller itself being initialized.
Alex or Marcel - since whatever it was I likely heard it from one of you
(or imagined it in a dream), can you straighten me out?
That only seems possible if we had a way to
signal phb hotplug *after* we've hotplugged some PCI devices on the bus,
which means we'd need some interface to trigger hotplug beyond the
standard 'device_add' calls, e.g.:
device_add spapr-pci-host-bridge,hotplug-deferred=true,id=phb2,index=2
device_add virtio-net-pci,bus=phb2.0,...,hotplug-deferred=true
device_signal_hotplug phb2
That's actually akin to how it's normally done on pHyp (not only for PHB
hotplug, but for PCI hotplug in general, which is why this could be
reasonably expected to work on pseries guests), but it seems quite a bit
different from how we'd normally handle this on kvm, which I think would
be something more like:
device_add spapr-pci-host-bridge,id=phb2,index=2
<wait for hotplug completion event>
device_add virtio-net-pci,bus=phb2.0,...
In which case it doesn't really matter if the guest scans the bus at
hotplug time or not. Is there some other scenario where this might
arise?
4:56 p.m.
On Thu, Jan 12, 2017 at 12:53:28PM -0500, Laine Stump wrote:
On 01/12/2017 11:35 AM, Michael Roth wrote:
> Quoting Laine Stump (2017-01-12 08:52:10)
> > On 01/12/2017 05:31 AM, Andrea Bolognani wrote:
> > > On Mon, 2017-01-09 at 10:46 +1100, David Gibson wrote:
> > > > > > * To allow for hotplugged devices, libvirt should also
add a number
> > > > > > of additional, empty vPHBs (the PAPR spec allows for
hotplug of
> > > > > > PHBs, but this is not yet implemented in qemu).
> > > > >
> > > > > "A number" here will have to mean "one",
same number of
> > > > > empty PCIe Root Ports libvirt will add to a newly-defined
> > > > > q35 guest.
> > > >
> > > > Umm.. why?
> > >
> > > Because some applications using libvirt would inevitably
> > > start relying on the fact that such spare PHBs are
> > > available, locking us into providing at least the same
> > > number forever. In other words, increasing the amount at
> > > a later time is always possible, but decreasing it isn't.
> > > We did the same when we started automatically adding PCIe
> > > Root Ports to q35 machines.
> > >
> > > The rationale is that having a single spare hotpluggable
> > > slot is extremely convenient for basic usage, eg. a simple
> > > guest created by someone who's not necessarily very
> > > familiar with virtualization; on the other hand, if you
> > > are actually deploying in production you ought to conduct
> > > proper capacity planning and figure out in advance how
> > > many devices you're likely to need to hotplug throughout
> > > the guest's life.
> >
> > And of course the reason we don't want to add "too many" extra
> > controllers by default is so that we don't end up with *all* guests
> > burdened with extra hardware they don't need or want. The libguestfs
> > appliance is one example of a libvirt consumer that definitely doesn't
> > want extra baggage in its guests - guest startup time is very important
> > to libguestfs, so any addition to the hardware list is looked upon with
> > disappointment.
> >
> > >
> > > Of course this all will be moot once we can hotplug PHBs :)
> >
> > Will the guest OSes handle that properly? I remember being told that
>
> I believe on pseries we *do* scan for devices on the PHB as part of
> bringing the PHB online in the hotplug path. But I'm not sure that
> matters (see below).
>
> > Linux, for example, doesn't scan the new bus for devices when a new
> > controller is added, making it pointless to hotplug a PCI controller (as
> > usual, it could be that I'm remembering incorrectly...)
> >
>
> Wouldn't that only be an issue if we hotplugged a PHB that already had
> PCI devices on the bus?
Yeah you're right, I'm probably remembering the wrong problem and wrong
reason for the problem. I just remember there was *some* issue about
hotplugging new PCI controllers. Possibly the internal representation of the
bus hierarchy wasn't updated unless you forced a rescan of all the devices
or something? My memory of it is vague, I just remember being told it wasn't
just a case of the controller itself being initialized.
Alex or Marcel - since whatever it was I likely heard it from one of you (or
imagined it in a dream), can you straighten me out?
Regardless, I'm pretty sure it won't be relevant for Power guests.
PHB hotplug has its own protocol in PAPR, and is used routinely for
Linux guests under PowerVM.
> That only seems possible if we had a way to
> signal phb hotplug *after* we've hotplugged some PCI devices on the bus,
> which means we'd need some interface to trigger hotplug beyond the
> standard 'device_add' calls, e.g.:
>
> device_add spapr-pci-host-bridge,hotplug-deferred=true,id=phb2,index=2
> device_add virtio-net-pci,bus=phb2.0,...,hotplug-deferred=true
> device_signal_hotplug phb2
>
> That's actually akin to how it's normally done on pHyp (not only for PHB
> hotplug, but for PCI hotplug in general, which is why this could be
> reasonably expected to work on pseries guests), but it seems quite a bit
> different from how we'd normally handle this on kvm, which I think would
> be something more like:
>
> device_add spapr-pci-host-bridge,id=phb2,index=2
> <wait for hotplug completion event>
> device_add virtio-net-pci,bus=phb2.0,...
>
> In which case it doesn't really matter if the guest scans the bus at
> hotplug time or not. Is there some other scenario where this might
> arise?
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
6:21 a.m.
On 01/12/2017 07:53 PM, Laine Stump wrote:
On 01/12/2017 11:35 AM, Michael Roth wrote:
> Quoting Laine Stump (2017-01-12 08:52:10)
[...]
Yeah you're right, I'm probably remembering the wrong problem and wrong reason
for the problem. I just remember there was *some* issue about hotplugging new PCI
controllers. Possibly the internal
representation of the bus hierarchy wasn't updated unless you forced a rescan of all
the devices or something? My memory of it is vague, I just remember being told it
wasn't just a case of the
controller itself being initialized.
Alex or Marcel - since whatever it was I likely heard it from one of you (or imagined it
in a dream), can you straighten me out?
Hi Laine,
Indeed, hot-plugging a QEMU pci-bridge in X86 is somehow problematic since
it comes with some bits on ACPI tables which are already loaded at hot-plug time.
We need the bits since the x86 pci-hotplug is ACPI based. For Q35 machine might be
easier,
but is not implemented yet as far as I know.
Thanks,
Marcel
[...]
4:57 p.m.
On Thu, Jan 12, 2017 at 11:31:35AM +0100, Andrea Bolognani wrote:
On Mon, 2017-01-09 at 10:46 +1100, David Gibson wrote:
> > > * To allow for hotplugged devices, libvirt should also add a number
> > > of additional, empty vPHBs (the PAPR spec allows for hotplug of
> > > PHBs, but this is not yet implemented in qemu).
> >
> > "A number" here will have to mean "one", same number of
> > empty PCIe Root Ports libvirt will add to a newly-defined
> > q35 guest.
>
> Umm.. why?
Because some applications using libvirt would inevitably
start relying on the fact that such spare PHBs are
available, locking us into providing at least the same
number forever. In other words, increasing the amount at
a later time is always possible, but decreasing it isn't.
We did the same when we started automatically adding PCIe
Root Ports to q35 machines.
The rationale is that having a single spare hotpluggable
slot is extremely convenient for basic usage, eg. a simple
guest created by someone who's not necessarily very
familiar with virtualization; on the other hand, if you
are actually deploying in production you ought to conduct
proper capacity planning and figure out in advance how
many devices you're likely to need to hotplug throughout
the guest's life.
Hm, ok. Well I guess the limitation is the same as on x86, so it
shouldn't surprise people.
Of course this all will be moot once we can hotplug PHBs :)
Yes. Unfortunately, nobody's actually working on that at present.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
10:29 a.m.
On Fri, 13 Jan 2017 09:57:36 +1100
David Gibson <david(a)gibson.dropbear.id.au> wrote:
On Thu, Jan 12, 2017 at 11:31:35AM +0100, Andrea Bolognani wrote:
> On Mon, 2017-01-09 at 10:46 +1100, David Gibson wrote:
> > > > * To allow for hotplugged devices, libvirt should also add a
number
> > > > of additional, empty vPHBs (the PAPR spec allows for hotplug of
> > > > PHBs, but this is not yet implemented in qemu).
> > >
> > > "A number" here will have to mean "one", same number
of
> > > empty PCIe Root Ports libvirt will add to a newly-defined
> > > q35 guest.
> >
> > Umm.. why?
>
> Because some applications using libvirt would inevitably
> start relying on the fact that such spare PHBs are
> available, locking us into providing at least the same
> number forever. In other words, increasing the amount at
> a later time is always possible, but decreasing it isn't.
> We did the same when we started automatically adding PCIe
> Root Ports to q35 machines.
>
> The rationale is that having a single spare hotpluggable
> slot is extremely convenient for basic usage, eg. a simple
> guest created by someone who's not necessarily very
> familiar with virtualization; on the other hand, if you
> are actually deploying in production you ought to conduct
> proper capacity planning and figure out in advance how
> many devices you're likely to need to hotplug throughout
> the guest's life.
Hm, ok. Well I guess the limitation is the same as on x86, so it
shouldn't surprise people.
> Of course this all will be moot once we can hotplug PHBs :)
Yes. Unfortunately, nobody's actually working on that at present.
Well, there might be someone now :)
Michael Roth had posted a RFC patchset back in 2015:
https://lists.gnu.org/archive/html/qemu-ppc/2015-04/msg00275.html
I'll start from here.
Cheers.
--
Greg
10:03 a.m.
On 01/05/2017 12:46 AM, David Gibson wrote:
There was a discussion back in November on the qemu list which
spilled
onto the libvirt list about how to add support for PCIe devices to
POWER VMs, specifically 'pseries' machine type PAPR guests.
Here's a more concrete proposal for how to handle part of this in
future from the libvirt side. Strictly speaking what I'm suggesting
here isn't intrinsically linked to PCIe: it will make adding PCIe
support sanely easier, as well as having a number of advantages for
both PCIe and plain-PCI devices on PAPR guests.
Background:
* Currently the pseries machine type only supports vanilla PCI
buses.
* This is a qemu limitation, not something inherent - PAPR guests
running under PowerVM (the IBM hypervisor) can use passthrough
PCIe devices (PowerVM doesn't emulate devices though).
* In fact the way PCI access is para-virtalized in PAPR makes the
usual distinctions between PCI and PCIe largely disappear
* Presentation of PCIe devices to PAPR guests is unusual
* Unlike x86 - and other "bare metal" platforms, root ports are
not made visible to the guest. i.e. all devices (typically)
appear as though they were integrated devices on x86
* In terms of topology all devices will appear in a way similar to
a vanilla PCI bus, even PCIe devices
* However PCIe extended config space is accessible
* This means libvirt's usual placement of PCIe devices is not
suitable for PAPR guests
* PAPR has its own hotplug mechanism
* This is used instead of standard PCIe hotplug
* This mechanism works for both PCIe and vanilla-PCI devices
* This can hotplug/unplug devices even without a root port P2P
bridge between it and the root "bus
* Multiple independent host bridges are routine on PAPR
* Unlike PC (where all host bridges have multiplexed access to
configuration space) PCI host bridges (PHBs) are truly
independent for PAPR guests (disjoint MMIO regions in system
address space)
* PowerVM typically presents a separate PHB to the guest for each
host slot passed through
The Proposal:
I suggest that libvirt implement a new default algorithm for placing
(i.e. assigning addresses to) both PCI and PCIe devices for (only)
PAPR guests.
The short summary is that by default it should assign each device to a
separate vPHB, creating vPHBs as necessary.
* For passthrough sometimes a group of host devices can't be safely
isolated from each other - this is known as a (host) Partitionable
Endpoint (PE). In this case, if any device in the PE is passed
through to a guest, the whole PE must be passed through to the
same vPHB in the guest. From the guest POV, each vPHB has exactly
one (guest) PE.
* To allow for hotplugged devices, libvirt should also add a number
of additional, empty vPHBs (the PAPR spec allows for hotplug of
PHBs, but this is not yet implemented in qemu). When hotplugging
a new device (or PE) libvirt should locate a vPHB which doesn't
currently contain anything.
* libvirt should only (automatically) add PHBs - never root ports or
other PCI to PCI bridges
It's a bit unconventional to leave all but one slot of a controller
unused, but your thinking makes sense. I don't think this will be as
large/disruptive of a change as you might be expecting - we already have
different addressing rules to automatically addressed vs. manually
addressed, as well as a framework in place to behave differently for
different PCI controllers (e.g. some support hotplug and others don't),
and to modify behavior based on machinetype / root bus model, so it
should be straightforward to do make things behave as you outline above.
(The first item in your list sounds exactly like VFIO iommu groups. Is
that how it's exposed on PPC? If so, libvirt already takes care of
guaranteeing that any devices in the same group aren't used by other
guests or the host during the time a guest is using a device. It doesn't
automatically assign the other devices to the guest though, since this
could have unexpected effects on host operation (the example that kept
coming up when this was originally discussed wrt vfio device assignment
was the case where a disk device in use on the host was attached to a
controller in the same iommu group as a USB controller that was going to
be assigned to a guest - silently assigning the disk controller to the
guest would cause the host's disk to suddenly become unusable).)
In order to handle migration, the vPHBs will need to be represented
in
the domain XML, which will also allow the user to override this
topology if they want.
Advantages:
There are still some details I need to figure out w.r.t. handling PCIe
devices (on both the qemu and libvirt sides). However the fact that
PAPR guests don't typically see PCIe root ports means that the normal
libvirt PCIe allocation scheme won't work.
Well, the "normal libvirt PCIe allocation scheme" assumes "normal
PCIe" :-).
This scheme has several
advantages with or without support for PCIe devices:
* Better performance for 32-bit devices
With multiple devices on a single vPHB they all must share a (fairly
small) 32-bit DMA/IOMMU window. With separate PHBs they each have a
separate window. PAPR guests have an always-on guest visible IOMMU.
* Better EEH handling for passthrough devices
EEH is an IBM hardware-assisted mechanism for isolating and safely
resetting devices experiencing hardware faults so they don't bring
down other devices or the system at large. It's roughly similar to
PCIe AER in concept, but has a different IBM specific interface, and
works on both PCI and PCIe devices.
Currently the kernel interfaces for handling EEH events on passthrough
devices will only work if there is a single (host) iommu group in the
vfio container. While lifting that restriction would be nice, it's
quite difficult to do so (it requires keeping state synchronized
between multiple host groups). That also means that an EEH error on
one device could stop another device where that isn't required by the
actual hardware.
The unit of EEH isolation is a PE (Partitionable Endpoint) and
currently there is only one guest PE per vPHB. Changing this might
also be possible, but is again quite complex and may result in
confusing and/or broken distinctions between groups for EEH isolation
and IOMMU isolation purposes.
Placing separate host groups in separate vPHBs sidesteps these
problems.
* Guest NUMA node assignment of devices
PAPR does not (and can't reasonably) use the pxb device. Instead to
allocate devices to different guest NUMA nodes they should be placed
on different vPHBs. Placing them on different PHBs by default allows
NUMA node to be assigned to those PHBs in a straightforward manner.
So far libvirt doesn't try to assign PCI addresses to devices according
to NUMA node, but assumes that the management application will manually
address devices that need to be put on a particular pxb (it's only since
the recent advent of the pxb that guests have become aware of multiple
NUMA nodes). Possibly in the future libvirt will attempt to
automatically place devices on a pxb that matches its NUMA node (if it
exists). We don't want to force use of pxb for all guests on a host that
has multiple NUMA nodes though. This might make more sense on PCC though
since all devices are on a PHB, and each PHB can have a NUMA node set.
5:14 p.m.
On Thu, Jan 12, 2017 at 11:03:05AM -0500, Laine Stump wrote:
On 01/05/2017 12:46 AM, David Gibson wrote:
> There was a discussion back in November on the qemu list which spilled
> onto the libvirt list about how to add support for PCIe devices to
> POWER VMs, specifically 'pseries' machine type PAPR guests.
>
> Here's a more concrete proposal for how to handle part of this in
> future from the libvirt side. Strictly speaking what I'm suggesting
> here isn't intrinsically linked to PCIe: it will make adding PCIe
> support sanely easier, as well as having a number of advantages for
> both PCIe and plain-PCI devices on PAPR guests.
>
> Background:
>
> * Currently the pseries machine type only supports vanilla PCI
> buses.
> * This is a qemu limitation, not something inherent - PAPR guests
> running under PowerVM (the IBM hypervisor) can use passthrough
> PCIe devices (PowerVM doesn't emulate devices though).
> * In fact the way PCI access is para-virtalized in PAPR makes the
> usual distinctions between PCI and PCIe largely disappear
> * Presentation of PCIe devices to PAPR guests is unusual
> * Unlike x86 - and other "bare metal" platforms, root ports are
> not made visible to the guest. i.e. all devices (typically)
> appear as though they were integrated devices on x86
> * In terms of topology all devices will appear in a way similar to
> a vanilla PCI bus, even PCIe devices
> * However PCIe extended config space is accessible
> * This means libvirt's usual placement of PCIe devices is not
> suitable for PAPR guests
> * PAPR has its own hotplug mechanism
> * This is used instead of standard PCIe hotplug
> * This mechanism works for both PCIe and vanilla-PCI devices
> * This can hotplug/unplug devices even without a root port P2P
> bridge between it and the root "bus
> * Multiple independent host bridges are routine on PAPR
> * Unlike PC (where all host bridges have multiplexed access to
> configuration space) PCI host bridges (PHBs) are truly
> independent for PAPR guests (disjoint MMIO regions in system
> address space)
> * PowerVM typically presents a separate PHB to the guest for each
> host slot passed through
>
> The Proposal:
>
> I suggest that libvirt implement a new default algorithm for placing
> (i.e. assigning addresses to) both PCI and PCIe devices for (only)
> PAPR guests.
>
> The short summary is that by default it should assign each device to a
> separate vPHB, creating vPHBs as necessary.
>
> * For passthrough sometimes a group of host devices can't be safely
> isolated from each other - this is known as a (host) Partitionable
> Endpoint (PE). In this case, if any device in the PE is passed
> through to a guest, the whole PE must be passed through to the
> same vPHB in the guest. From the guest POV, each vPHB has exactly
> one (guest) PE.
> * To allow for hotplugged devices, libvirt should also add a number
> of additional, empty vPHBs (the PAPR spec allows for hotplug of
> PHBs, but this is not yet implemented in qemu). When hotplugging
> a new device (or PE) libvirt should locate a vPHB which doesn't
> currently contain anything.
> * libvirt should only (automatically) add PHBs - never root ports or
> other PCI to PCI bridges
It's a bit unconventional to leave all but one slot of a controller unused,
Unconventional for x86, maybe. It's been SOP on IBM Power for a
decade or more. Both for PAPR guests and in some cases on the
physical hardware (AIUI many, though not all, Power systems used a
separate host bridge for each physical slot to ensure better isolation
between devices).
but your thinking makes sense. I don't think this will be as
large/disruptive of a change as you might be expecting - we already have
different addressing rules to automatically addressed vs. manually
addressed, as well as a framework in place to behave differently for
different PCI controllers (e.g. some support hotplug and others don't), and
to modify behavior based on machinetype / root bus model, so it should be
straightforward to do make things behave as you outline above.
Actually, I had that impression, so I was hoping it wouldn't be too
bad to implement. I'd really like to get this underway ASAP, so we
can build the PCIe support (both qemu and Power) around that.
(The first item in your list sounds exactly like VFIO iommu groups.
Is that
how it's exposed on PPC?
Yes, for Power hosts and guests there's a 1-1 correspondance between
PEs and IOMMU groups. Technically speaking, I believe the PE provides
more isolation guarantees than the IOMMU group, but they're generally
close enough in practice.
If so, libvirt already takes care of guaranteeing
that any devices in the same group aren't used by other guests or the host
during the time a guest is using a device.
Yes, I'm aware of that, that's not an aspect I was concerned about.
Although that said, last I heard there was a bug in libvirt which
on hot *un*plug could assign devices back to the host without waiting
for other assigned devices in the group, which can crash the host.
It doesn't automatically assign
the other devices to the guest though, since this could have unexpected
effects on host operation (the example that kept coming up when this was
originally discussed wrt vfio device assignment was the case where a disk
device in use on the host was attached to a controller in the same iommu
group as a USB controller that was going to be assigned to a guest -
silently assigning the disk controller to the guest would cause the host's
disk to suddenly become unusable).)
Um.. what!? If something in the group is assigned to the guest, other
devices in the group MUST NOT be used by the hast, regardless of
whether they are actually assigned to the guest or not. In the
situation you describe the guest would control the IOMMU mappings for
the host's disk device making it no more usable, and a whole lot more
dangerous.
In fact I'm pretty sure VFIO won't let you do that: it won't let you
add the group to a VFIO container until all devices in the group are
bound to the VFIO stub driver instead of whatever host driver they
were using before.
> In order to handle migration, the vPHBs will need to be
represented in
> the domain XML, which will also allow the user to override this
> topology if they want.
>
> Advantages:
>
> There are still some details I need to figure out w.r.t. handling PCIe
> devices (on both the qemu and libvirt sides). However the fact that
> PAPR guests don't typically see PCIe root ports means that the normal
> libvirt PCIe allocation scheme won't work.
Well, the "normal libvirt PCIe allocation scheme" assumes "normal
PCIe" :-).
My point exactly.
> This scheme has several
> advantages with or without support for PCIe devices:
>
> * Better performance for 32-bit devices
>
> With multiple devices on a single vPHB they all must share a (fairly
> small) 32-bit DMA/IOMMU window. With separate PHBs they each have a
> separate window. PAPR guests have an always-on guest visible IOMMU.
>
> * Better EEH handling for passthrough devices
>
> EEH is an IBM hardware-assisted mechanism for isolating and safely
> resetting devices experiencing hardware faults so they don't bring
> down other devices or the system at large. It's roughly similar to
> PCIe AER in concept, but has a different IBM specific interface, and
> works on both PCI and PCIe devices.
>
> Currently the kernel interfaces for handling EEH events on passthrough
> devices will only work if there is a single (host) iommu group in the
> vfio container. While lifting that restriction would be nice, it's
> quite difficult to do so (it requires keeping state synchronized
> between multiple host groups). That also means that an EEH error on
> one device could stop another device where that isn't required by the
> actual hardware.
>
> The unit of EEH isolation is a PE (Partitionable Endpoint) and
> currently there is only one guest PE per vPHB. Changing this might
> also be possible, but is again quite complex and may result in
> confusing and/or broken distinctions between groups for EEH isolation
> and IOMMU isolation purposes.
>
> Placing separate host groups in separate vPHBs sidesteps these
> problems.
>
> * Guest NUMA node assignment of devices
>
> PAPR does not (and can't reasonably) use the pxb device. Instead to
> allocate devices to different guest NUMA nodes they should be placed
> on different vPHBs. Placing them on different PHBs by default allows
> NUMA node to be assigned to those PHBs in a straightforward manner.
So far libvirt doesn't try to assign PCI addresses to devices according to
NUMA node, but assumes that the management application will manually address
devices that need to be put on a particular pxb (it's only since the recent
advent of the pxb that guests have become aware of multiple NUMA nodes).
Possibly in the future libvirt will attempt to automatically place devices
on a pxb that matches its NUMA node (if it exists). We don't want to force
use of pxb for all guests on a host that has multiple NUMA nodes though.
This might make more sense on PCC though since all devices are on a PHB, and
each PHB can have a NUMA node set.
I hope the connection of pxb to NUMA allocation isn't too tight in
libvirt. pxb is essentially an x86 specific hack. For PAPR guests
the correct way to assign a NUMA node to PCI devices is to put them on
separate vPHBs and set the node of the vPHB. The point above is
noting that once this proposal is implemented, all we need to do to
add NUMA awareness is allow a NUMA node property on the vPHBs.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
6:27 a.m.
On 01/13/2017 01:14 AM, David Gibson wrote:
On Thu, Jan 12, 2017 at 11:03:05AM -0500, Laine Stump wrote:
> On 01/05/2017 12:46 AM, David Gibson wrote:
>> There was a discussion back in November on the qemu list which spilled
>> onto the libvirt list about how to add support for PCIe devices to
>> POWER VMs, specifically 'pseries' machine type PAPR guests.
>>
>> Here's a more concrete proposal for how to handle part of this in
>> future from the libvirt side. Strictly speaking what I'm suggesting
>> here isn't intrinsically linked to PCIe: it will make adding PCIe
>> support sanely easier, as well as having a number of advantages for
>> both PCIe and plain-PCI devices on PAPR guests.
>>
>> Background:
>>
>> * Currently the pseries machine type only supports vanilla PCI
>> buses.
>> * This is a qemu limitation, not something inherent - PAPR guests
>> running under PowerVM (the IBM hypervisor) can use passthrough
>> PCIe devices (PowerVM doesn't emulate devices though).
>> * In fact the way PCI access is para-virtalized in PAPR makes the
>> usual distinctions between PCI and PCIe largely disappear
>> * Presentation of PCIe devices to PAPR guests is unusual
>> * Unlike x86 - and other "bare metal" platforms, root ports are
>> not made visible to the guest. i.e. all devices (typically)
>> appear as though they were integrated devices on x86
>> * In terms of topology all devices will appear in a way similar to
>> a vanilla PCI bus, even PCIe devices
>> * However PCIe extended config space is accessible
>> * This means libvirt's usual placement of PCIe devices is not
>> suitable for PAPR guests
>> * PAPR has its own hotplug mechanism
>> * This is used instead of standard PCIe hotplug
>> * This mechanism works for both PCIe and vanilla-PCI devices
>> * This can hotplug/unplug devices even without a root port P2P
>> bridge between it and the root "bus
>> * Multiple independent host bridges are routine on PAPR
>> * Unlike PC (where all host bridges have multiplexed access to
>> configuration space) PCI host bridges (PHBs) are truly
>> independent for PAPR guests (disjoint MMIO regions in system
>> address space)
>> * PowerVM typically presents a separate PHB to the guest for each
>> host slot passed through
>>
>> The Proposal:
>>
>> I suggest that libvirt implement a new default algorithm for placing
>> (i.e. assigning addresses to) both PCI and PCIe devices for (only)
>> PAPR guests.
>>
>> The short summary is that by default it should assign each device to a
>> separate vPHB, creating vPHBs as necessary.
>>
>> * For passthrough sometimes a group of host devices can't be safely
>> isolated from each other - this is known as a (host) Partitionable
>> Endpoint (PE). In this case, if any device in the PE is passed
>> through to a guest, the whole PE must be passed through to the
>> same vPHB in the guest. From the guest POV, each vPHB has exactly
>> one (guest) PE.
>> * To allow for hotplugged devices, libvirt should also add a number
>> of additional, empty vPHBs (the PAPR spec allows for hotplug of
>> PHBs, but this is not yet implemented in qemu). When hotplugging
>> a new device (or PE) libvirt should locate a vPHB which doesn't
>> currently contain anything.
>> * libvirt should only (automatically) add PHBs - never root ports or
>> other PCI to PCI bridges
>
>
> It's a bit unconventional to leave all but one slot of a controller unused,
Unconventional for x86, maybe. It's been SOP on IBM Power for a
decade or more. Both for PAPR guests and in some cases on the
physical hardware (AIUI many, though not all, Power systems used a
separate host bridge for each physical slot to ensure better isolation
between devices).
> but your thinking makes sense. I don't think this will be as
> large/disruptive of a change as you might be expecting - we already have
> different addressing rules to automatically addressed vs. manually
> addressed, as well as a framework in place to behave differently for
> different PCI controllers (e.g. some support hotplug and others don't), and
> to modify behavior based on machinetype / root bus model, so it should be
> straightforward to do make things behave as you outline above.
Actually, I had that impression, so I was hoping it wouldn't be too
bad to implement. I'd really like to get this underway ASAP, so we
can build the PCIe support (both qemu and Power) around that.
> (The first item in your list sounds exactly like VFIO iommu groups. Is that
> how it's exposed on PPC?
Yes, for Power hosts and guests there's a 1-1 correspondance between
PEs and IOMMU groups. Technically speaking, I believe the PE provides
more isolation guarantees than the IOMMU group, but they're generally
close enough in practice.
> If so, libvirt already takes care of guaranteeing
> that any devices in the same group aren't used by other guests or the host
> during the time a guest is using a device.
Yes, I'm aware of that, that's not an aspect I was concerned about.
Although that said, last I heard there was a bug in libvirt which
on hot *un*plug could assign devices back to the host without waiting
for other assigned devices in the group, which can crash the host.
> It doesn't automatically assign
> the other devices to the guest though, since this could have unexpected
> effects on host operation (the example that kept coming up when this was
> originally discussed wrt vfio device assignment was the case where a disk
> device in use on the host was attached to a controller in the same iommu
> group as a USB controller that was going to be assigned to a guest -
> silently assigning the disk controller to the guest would cause the host's
> disk to suddenly become unusable).)
Um.. what!? If something in the group is assigned to the guest, other
devices in the group MUST NOT be used by the hast, regardless of
whether they are actually assigned to the guest or not. In the
situation you describe the guest would control the IOMMU mappings for
the host's disk device making it no more usable, and a whole lot more
dangerous.
In fact I'm pretty sure VFIO won't let you do that: it won't let you
add the group to a VFIO container until all devices in the group are
bound to the VFIO stub driver instead of whatever host driver they
were using before.
>> In order to handle migration, the vPHBs will need to be represented in
>> the domain XML, which will also allow the user to override this
>> topology if they want.
>>
>> Advantages:
>>
>> There are still some details I need to figure out w.r.t. handling PCIe
>> devices (on both the qemu and libvirt sides). However the fact that
>> PAPR guests don't typically see PCIe root ports means that the normal
>> libvirt PCIe allocation scheme won't work.
>
>
> Well, the "normal libvirt PCIe allocation scheme" assumes "normal
PCIe" :-).
My point exactly.
>
>
>> This scheme has several
>> advantages with or without support for PCIe devices:
>>
>> * Better performance for 32-bit devices
>>
>> With multiple devices on a single vPHB they all must share a (fairly
>> small) 32-bit DMA/IOMMU window. With separate PHBs they each have a
>> separate window. PAPR guests have an always-on guest visible IOMMU.
>>
>> * Better EEH handling for passthrough devices
>>
>> EEH is an IBM hardware-assisted mechanism for isolating and safely
>> resetting devices experiencing hardware faults so they don't bring
>> down other devices or the system at large. It's roughly similar to
>> PCIe AER in concept, but has a different IBM specific interface, and
>> works on both PCI and PCIe devices.
>>
>> Currently the kernel interfaces for handling EEH events on passthrough
>> devices will only work if there is a single (host) iommu group in the
>> vfio container. While lifting that restriction would be nice, it's
>> quite difficult to do so (it requires keeping state synchronized
>> between multiple host groups). That also means that an EEH error on
>> one device could stop another device where that isn't required by the
>> actual hardware.
>>
>> The unit of EEH isolation is a PE (Partitionable Endpoint) and
>> currently there is only one guest PE per vPHB. Changing this might
>> also be possible, but is again quite complex and may result in
>> confusing and/or broken distinctions between groups for EEH isolation
>> and IOMMU isolation purposes.
>>
>> Placing separate host groups in separate vPHBs sidesteps these
>> problems.
>>
>> * Guest NUMA node assignment of devices
>>
>> PAPR does not (and can't reasonably) use the pxb device. Instead to
>> allocate devices to different guest NUMA nodes they should be placed
>> on different vPHBs. Placing them on different PHBs by default allows
>> NUMA node to be assigned to those PHBs in a straightforward manner.
>
> So far libvirt doesn't try to assign PCI addresses to devices according to
> NUMA node, but assumes that the management application will manually address
> devices that need to be put on a particular pxb (it's only since the recent
> advent of the pxb that guests have become aware of multiple NUMA nodes).
> Possibly in the future libvirt will attempt to automatically place devices
> on a pxb that matches its NUMA node (if it exists). We don't want to force
> use of pxb for all guests on a host that has multiple NUMA nodes though.
> This might make more sense on PCC though since all devices are on a PHB, and
> each PHB can have a NUMA node set.
I hope the connection of pxb to NUMA allocation isn't too tight in
libvirt. pxb is essentially an x86 specific hack. For PAPR guests
the correct way to assign a NUMA node to PCI devices is to put them on
separate vPHBs and set the node of the vPHB. The point above is
noting that once this proposal is implemented, all we need to do to
add NUMA awareness is allow a NUMA node property on the vPHBs.
The X86/(ACPI based ARM?) pxb works exactly like that.
It has a numa_nr property that will associate the PXB to a guest NUMA node.
All devices behind that pxb are connected to pxb's NUMA node.
Thanks,
Marcel
2642
days inactive
2691
days old
26 comments
7 participants
participants (7)
-
Alexey Kardashevskiy -
Andrea Bolognani -
David Gibson -
Greg Kurz -
Laine Stump -
Marcel Apfelbaum -
Michael Roth