1:13 p.m.
Now that qemu is getting the q35 machine type, libvirt needs to support it.
As far as I understand, from libvirt's point of view, q35 is just another x86_64
system, but with a different set of implicit devices, and possibly some extra rules
limiting which devices can be plugged into which bus/slot on the guest. That means that in
order to support it, we will need to recognize when a q35-based machine type is being
created, and auto-add all the implicit devices to libvirt's config model for that
domain, then pay attention to the extra rules when assigning addresses for all the
user-added devices.
We already add implicit controllers/devices for pc-based machine types; as a matter of
fact, currently, libvirt improperly assumes (for the purposes of adding implicit devices)
that *every* virtual machine is based on the "pc" machine type (or rather it
just doesn't pay attention), so it always adds all the implicit devices for a pc
machine type for every domain. This of course is already incorrect for many (probably
all?) non-x86 machine types, even before we add q35 into the mix. To fix this, it might be
reasonable (and arguably, it's necessary to fix the problem in a backward-compatible
manner) to just setup a table of machinetype ==> implicit device lists, look up the
machine type in this table, and add the devices needed for that machine type. This goes
against libvirt's longstanding view of machinetype as being an opaque value that it
merely passes through to qemu, but it's manageable for the existing machine types
(even including q35), since it's a finite set. But it starts to be a pain to maintain
when you think about future additions - yet another case where new functionality in qemu
will require an update to libvirt before it can be fully used via libvirt.
In the long term, it would be very useful / more easily maintainable to have a qemu status
command available via QMP which would return the list of implicit devices (and their PCI
addresses) for any requested machine type. It would be necessary that this command be
callable multiple times within a single execution of qemu, giving it a different
machinetype each time. This way libvirt could first query the list of available
machinetypes in this particular qemu binary, then request the list of implicit devices for
each machine type (libvirt runs each available qemu binary *once* the first time it's
requested, and caches all such capabilities information so that it doesn't need to
re-run qemu again and again). My limited understanding of qemu's code is that qemu
itself doesn't have a table of this information as data, but instead has lines of code
that are executed to create it, thus making it impractical to provide the list of devices
for a machinetype without actually instantiating a machine of that type. What's the
feasibility of adding such a capability (and in the process likely making the list of
implicit devices in qemu itself table/data driven rather than constructed with lines of
code).
More questions:
1) It seems that the exact list of devices for the basic q35 machine type hasn't been
settled on yet, is that correct?
2) Are there other issues aside from implicit controller devices I need to consider for
q35? For example, are there any devices that (as I recall is the case for some devices on
"pc") may or may not be present, but if they are present they are always at a
particular PCI address (meaning that address must be reserved)? I've also just learned
that certain types of PCIe devices must be plugged into certain locations on the guest
bus? ("root complex" devices - is there a good source of background info to
learn the meaning of terms like that, and the rules of engagement? libvirt will need to
know/follow these rules.)
3) What new types of devices/controllers must be supported for a properly functioning q35
machine?
2:15 p.m.
New subject: [libvirt] [Qemu-devel] q35 machine type and libvirt.
On Wed, 2013-02-06 at 14:13 -0500, Laine Stump wrote:
Now that qemu is getting the q35 machine type, libvirt needs to
support it.
As far as I understand, from libvirt's point of view, q35 is just
another x86_64 system, but with a different set of implicit devices,
and possibly some extra rules limiting which devices can be plugged
into which bus/slot on the guest. That means that in order to support
it, we will need to recognize when a q35-based machine type is being
created, and auto-add all the implicit devices to libvirt's config
model for that domain, then pay attention to the extra rules when
assigning addresses for all the user-added devices.
We already add implicit controllers/devices for pc-based machine
types; as a matter of fact, currently, libvirt improperly assumes (for
the purposes of adding implicit devices) that *every* virtual machine
is based on the "pc" machine type (or rather it just doesn't pay
attention), so it always adds all the implicit devices for a pc
machine type for every domain. This of course is already incorrect for
many (probably all?) non-x86 machine types, even before we add q35
into the mix. To fix this, it might be reasonable (and arguably, it's
necessary to fix the problem in a backward-compatible manner) to just
setup a table of machinetype ==> implicit device lists, look up the
machine type in this table, and add the devices needed for that
machine type. This goes against libvirt's longstanding view of
machinetype as being an opaque value that it merely passes through to
qemu, but it's manageable for the existing machine types (even
including q35), since it's a finite set. But it starts to be a pain to
maintain when you think about future additions - yet another case
where new functionality in qemu will require an update to libvirt
before it can be fully used via libvirt.
In the long term, it would be very useful / more easily maintainable
to have a qemu status command available via QMP which would return the
list of implicit devices (and their PCI addresses) for any requested
machine type. It would be necessary that this command be callable
multiple times within a single execution of qemu, giving it a
different machinetype each time. This way libvirt could first query
the list of available machinetypes in this particular qemu binary,
then request the list of implicit devices for each machine type
(libvirt runs each available qemu binary *once* the first time it's
requested, and caches all such capabilities information so that it
doesn't need to re-run qemu again and again). My limited understanding
of qemu's code is that qemu itself doesn't have a table of this
information as data, but instead has lines of code that are executed
to create it, thus making it impractical to provide the list of
devices for a machinetype without actually instantiating a machine of
that type. What's the feasibility of adding such a capability (and in
the process likely making the list of implicit devices in qemu itself
table/data driven rather than constructed with lines of code).
More questions:
1) It seems that the exact list of devices for the basic q35 machine
type hasn't been settled on yet, is that correct?
I think what we have currently is just a stepping stone to a base
configuration. At a minimum, we're missing the PCI bridge attached to
the ICH, which is where I think libvirt should attach non-chipset
component devices. Next would be PCIe root ports where emulated and
assigned PCIe devices could be attached.
2) Are there other issues aside from implicit controller devices I
need to consider for q35? For example, are there any devices that (as
I recall is the case for some devices on "pc") may or may not be
present, but if they are present they are always at a particular PCI
address (meaning that address must be reserved)? I've also just
learned that certain types of PCIe devices must be plugged into
certain locations on the guest bus? ("root complex" devices - is there
a good source of background info to learn the meaning of terms like
that, and the rules of engagement? libvirt will need to know/follow
these rules.)
The GMCH (Graphics & Memory Controller Hub) defines:
00.0 - Host bridge
01.0 - x16 root port for external graphics
02.0,1 - integrated graphics device (IGD)
03.0,1,2,3 - management engine subsystem
And the ICH defines:
19.0 - Embedded ethernet (e1000e)
1a.* - UHCI/EHCI
1b.0 - HDA audio
1c.* - PCIe root ports
1d.* - UHCI/EHCI
1e.0 - PCI Bridge
1f.0 - ISA Bridge
1f.2,5 - SATA
1f.3 - SMBUS
Personally, I think these slots should be reserved for only the spec
defined devices, and I'm not all that keen on using the remaining slots
for anything else. Users should of course be allowed to put anything
anywhere, but libvirt auto-placement should follow some rules.
All of the above sit on what we now call bus pcie.0. This is a root
complex, which implies that all of endpoints are root complex integrated
endpoints. Being an integrated endpoint restricts aspects of the
device. I've already found out the hard way that Windows actually cares
about this and will ignore PCI assigned devices of type "Endpoint" when
attached to the root complex bus. (endpoint, root complex, etc is
defined in the PCIe spec, the above slot use is defined in the
respective chipset spec)
What I'd like to see is to implement the PCI-bridge at 1e.0 to expose a
complete, virgin PCI bus. libvirt should use that as the default
location for any PCI device that's not a chipset component. We might be
able to get away with installing our e1000 at 19.0, but otherwise I'm
thinking that the list only includes uhci/ehci, hda, ahci, and the
chipset components themselves (smbus, isa, root ports, etc...). We
don't have "IGD", so our graphics should go on the PCI bus and the PCI
bridge should include functioning VGA enable bits. Maybe QXL wants to
make itself a PCIe device, in which case it should be attached behind a
PCIe root port at slot 01.0. Secondary PCIe graphics attach to root
ports behind 1c.*. This is the same framework within real hardware has
to work.
Assigned devices get interesting due to the PCIe type. We've never had
any problems attaching PCIe devices to PCI buses on PIIX (but it may be
holding back our ability to support graphics passthrough), so assigned
devices can probably be attached to the PCI bus. More appropriate would
be to attach "Endpoints" behind root ports and "Integrated Endpoints"
to
the root complex. I've got some code that will mangle the PCIe type to
it's location in the topology, but it needs more work. That should help
make things more flexible.
3) What new types of devices/controllers must be supported for a
properly functioning q35 machine?
AHCI, bridges, root ports (we can skip these w/o PCIe devices, but for
hotplug we might want them fully populated - otherwise everything gets
hotplugged to the PCI bus). Thanks,
Alex
11:31 a.m.
New subject: [libvirt] [Qemu-devel] q35 machine type and libvirt.
On Wed, Feb 06, 2013 at 01:15:05PM -0700, Alex Williamson wrote:
On Wed, 2013-02-06 at 14:13 -0500, Laine Stump wrote:
> 2) Are there other issues aside from implicit controller devices I
> need to consider for q35? For example, are there any devices that (as
> I recall is the case for some devices on "pc") may or may not be
> present, but if they are present they are always at a particular PCI
> address (meaning that address must be reserved)? I've also just
> learned that certain types of PCIe devices must be plugged into
> certain locations on the guest bus? ("root complex" devices - is there
> a good source of background info to learn the meaning of terms like
> that, and the rules of engagement? libvirt will need to know/follow
> these rules.)
The GMCH (Graphics & Memory Controller Hub) defines:
00.0 - Host bridge
01.0 - x16 root port for external graphics
02.0,1 - integrated graphics device (IGD)
03.0,1,2,3 - management engine subsystem
And the ICH defines:
19.0 - Embedded ethernet (e1000e)
1a.* - UHCI/EHCI
1b.0 - HDA audio
1c.* - PCIe root ports
1d.* - UHCI/EHCI
1e.0 - PCI Bridge
1f.0 - ISA Bridge
1f.2,5 - SATA
1f.3 - SMBUS
Personally, I think these slots should be reserved for only the spec
defined devices, and I'm not all that keen on using the remaining slots
for anything else. Users should of course be allowed to put anything
anywhere, but libvirt auto-placement should follow some rules.
All of the above sit on what we now call bus pcie.0. This is a root
complex, which implies that all of endpoints are root complex integrated
endpoints. Being an integrated endpoint restricts aspects of the
device. I've already found out the hard way that Windows actually cares
about this and will ignore PCI assigned devices of type "Endpoint" when
attached to the root complex bus. (endpoint, root complex, etc is
defined in the PCIe spec, the above slot use is defined in the
respective chipset spec)
What I'd like to see is to implement the PCI-bridge at 1e.0 to expose a
complete, virgin PCI bus. libvirt should use that as the default
location for any PCI device that's not a chipset component. We might be
able to get away with installing our e1000 at 19.0, but otherwise I'm
thinking that the list only includes uhci/ehci, hda, ahci, and the
chipset components themselves (smbus, isa, root ports, etc...). We
don't have "IGD", so our graphics should go on the PCI bus and the PCI
bridge should include functioning VGA enable bits. Maybe QXL wants to
make itself a PCIe device, in which case it should be attached behind a
PCIe root port at slot 01.0. Secondary PCIe graphics attach to root
ports behind 1c.*. This is the same framework within real hardware has
to work.
Assigned devices get interesting due to the PCIe type. We've never had
any problems attaching PCIe devices to PCI buses on PIIX (but it may be
holding back our ability to support graphics passthrough), so assigned
devices can probably be attached to the PCI bus. More appropriate would
be to attach "Endpoints" behind root ports and "Integrated Endpoints"
to
the root complex. I've got some code that will mangle the PCIe type to
it's location in the topology, but it needs more work. That should help
make things more flexible.
So taking all this into account there are a couple of pieces of info
libvirt will need to know in order to assign device addresses / buses
sensibly
- What devices / buses are hardcoded to be always present (and their
addresses)
- What extra "integrated" devices are available for optional
enablement and their mandatory addresses (if any)
- What bus any other devices should be placed on by default.
With this in mind, libvirt's address assignment code is basically
already broken for anything which isn't a x86 system with PIIX
controller. eg address assignment for QEMU ARM / PPC / etc is
mostly fubar. Regardless of what Q35 involves, libvirt needs to
sort out the existing mess it has in this area. Sorting this
should then make support for Q35 more or less trivial, since all
the hardwork will have been done.
Since it doesn't sound like QEMU has a pratical means to supply
the data required, without actually running the machine, there
are only two options I see
- Libvirt maintains a set of data tables with all the info for
all QEMU machine types, in all system emulators. This will
need updating as QEMU gains new machine types.
- Libvirt tries to configure the VM without any addresses,
then launches QEMU and tries to introspect it to figure out
what QEMU assigned. Then shut it down, record the addresses
in XML for later usage at the real startup point.
Neither of these are particularly appealing to me, but I have a
preference for recording a data tables, since it would be much
simpler than trying to introspect things & more efficient too.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
12:13 p.m.
New subject: [libvirt] [Qemu-devel] q35 machine type and libvirt.
On Thu, 2013-02-07 at 17:31 +0000, Daniel P. Berrange wrote:
On Wed, Feb 06, 2013 at 01:15:05PM -0700, Alex Williamson wrote:
> On Wed, 2013-02-06 at 14:13 -0500, Laine Stump wrote:
> > 2) Are there other issues aside from implicit controller devices I
> > need to consider for q35? For example, are there any devices that (as
> > I recall is the case for some devices on "pc") may or may not be
> > present, but if they are present they are always at a particular PCI
> > address (meaning that address must be reserved)? I've also just
> > learned that certain types of PCIe devices must be plugged into
> > certain locations on the guest bus? ("root complex" devices - is
there
> > a good source of background info to learn the meaning of terms like
> > that, and the rules of engagement? libvirt will need to know/follow
> > these rules.)
>
> The GMCH (Graphics & Memory Controller Hub) defines:
>
> 00.0 - Host bridge
> 01.0 - x16 root port for external graphics
> 02.0,1 - integrated graphics device (IGD)
> 03.0,1,2,3 - management engine subsystem
>
> And the ICH defines:
>
> 19.0 - Embedded ethernet (e1000e)
> 1a.* - UHCI/EHCI
> 1b.0 - HDA audio
> 1c.* - PCIe root ports
> 1d.* - UHCI/EHCI
> 1e.0 - PCI Bridge
> 1f.0 - ISA Bridge
> 1f.2,5 - SATA
> 1f.3 - SMBUS
>
> Personally, I think these slots should be reserved for only the spec
> defined devices, and I'm not all that keen on using the remaining slots
> for anything else. Users should of course be allowed to put anything
> anywhere, but libvirt auto-placement should follow some rules.
>
> All of the above sit on what we now call bus pcie.0. This is a root
> complex, which implies that all of endpoints are root complex integrated
> endpoints. Being an integrated endpoint restricts aspects of the
> device. I've already found out the hard way that Windows actually cares
> about this and will ignore PCI assigned devices of type "Endpoint" when
> attached to the root complex bus. (endpoint, root complex, etc is
> defined in the PCIe spec, the above slot use is defined in the
> respective chipset spec)
>
> What I'd like to see is to implement the PCI-bridge at 1e.0 to expose a
> complete, virgin PCI bus. libvirt should use that as the default
> location for any PCI device that's not a chipset component. We might be
> able to get away with installing our e1000 at 19.0, but otherwise I'm
> thinking that the list only includes uhci/ehci, hda, ahci, and the
> chipset components themselves (smbus, isa, root ports, etc...). We
> don't have "IGD", so our graphics should go on the PCI bus and the
PCI
> bridge should include functioning VGA enable bits. Maybe QXL wants to
> make itself a PCIe device, in which case it should be attached behind a
> PCIe root port at slot 01.0. Secondary PCIe graphics attach to root
> ports behind 1c.*. This is the same framework within real hardware has
> to work.
>
> Assigned devices get interesting due to the PCIe type. We've never had
> any problems attaching PCIe devices to PCI buses on PIIX (but it may be
> holding back our ability to support graphics passthrough), so assigned
> devices can probably be attached to the PCI bus. More appropriate would
> be to attach "Endpoints" behind root ports and "Integrated
Endpoints" to
> the root complex. I've got some code that will mangle the PCIe type to
> it's location in the topology, but it needs more work. That should help
> make things more flexible.
So taking all this into account there are a couple of pieces of info
libvirt will need to know in order to assign device addresses / buses
sensibly
- What devices / buses are hardcoded to be always present (and their
addresses)
- What extra "integrated" devices are available for optional
enablement and their mandatory addresses (if any)
- What bus any other devices should be placed on by default.
With this in mind, libvirt's address assignment code is basically
already broken for anything which isn't a x86 system with PIIX
controller. eg address assignment for QEMU ARM / PPC / etc is
mostly fubar. Regardless of what Q35 involves, libvirt needs to
sort out the existing mess it has in this area. Sorting this
should then make support for Q35 more or less trivial, since all
the hardwork will have been done.
Since it doesn't sound like QEMU has a pratical means to supply
the data required, without actually running the machine, there
are only two options I see
- Libvirt maintains a set of data tables with all the info for
all QEMU machine types, in all system emulators. This will
need updating as QEMU gains new machine types.
- Libvirt tries to configure the VM without any addresses,
then launches QEMU and tries to introspect it to figure out
what QEMU assigned. Then shut it down, record the addresses
in XML for later usage at the real startup point.
Neither of these are particularly appealing to me, but I have a
preference for recording a data tables, since it would be much
simpler than trying to introspect things & more efficient too.
It doesn't seem like introspection really works unless we have a fully
populated base configuration for you to parse. Even then I'm not sure
how you'd figure out whether you should be placing devices to fill the
gaps on a given bus or not. -M q35 as we have it today is just a bare
minimum shell which doesn't tell much of anything on inspection. You
need the blueprint derived from the chipset spec of how to put all the
other components together, which seems more like the data tables you
describe. Thanks,
Alex
2:22 a.m.
New subject: [libvirt] [Qemu-devel] q35 machine type and libvirt.
Cc'ing a few QOMmers...
Laine Stump <laine(a)redhat.com> writes:
Now that qemu is getting the q35 machine type, libvirt needs to
support it.
As far as I understand, from libvirt's point of view, q35 is just
another x86_64 system, but with a different set of implicit devices,
and possibly some extra rules limiting which devices can be plugged
into which bus/slot on the guest. That means that in order to support
it, we will need to recognize when a q35-based machine type is being
created, and auto-add all the implicit devices to libvirt's config
model for that domain, then pay attention to the extra rules when
assigning addresses for all the user-added devices.
We already add implicit controllers/devices for pc-based machine
types; as a matter of fact, currently, libvirt improperly assumes (for
the purposes of adding implicit devices) that *every* virtual machine
is based on the "pc" machine type (or rather it just doesn't pay
attention), so it always adds all the implicit devices for a pc
machine type for every domain. This of course is already incorrect for
many (probably all?) non-x86 machine types, even before we add q35
into the mix. To fix this, it might be reasonable (and arguably, it's
necessary to fix the problem in a backward-compatible manner) to just
setup a table of machinetype ==> implicit device lists, look up the
machine type in this table, and add the devices needed for that
machine type. This goes against libvirt's longstanding view of
machinetype as being an opaque value that it merely passes through to
qemu, but it's manageable for the existing machine types (even
including q35), since it's a finite set. But it starts to be a pain to
maintain when you think about future additions - yet another case
where new functionality in qemu will require an update to libvirt
before it can be fully used via libvirt.
In the long term, it would be very useful / more easily maintainable
to have a qemu status command available via QMP which would return the
list of implicit devices (and their PCI addresses) for any requested
machine type.
You want to ask QEMU to describe the "board", i.e. the machine without
any optional devices.
The way you can do that now is to create a machine without any optional
devices, then introspect.
Introspection tools:
* info qtree (HMP only)
Mentioned for completeness. You really want QMP here.
* qom-list and qom-get (QMP only)
These let you examine QOM as attributed graph.
Non-qdevified devices are invisible. Show stopper only when this
includes devices libvirt needs to know about. For instance, a
mandatory device can be safely ignored as long as its resources cannot
conflict with anything libvirt might want to plug in.
Ceterum censeo we need full qdevification
It would be necessary that this command be callable
multiple times within a single execution of qemu, giving it a
different machinetype each time.
This isn't feasible without major surgery, as far as I can tell.
This way libvirt could first query
the list of available machinetypes in this particular qemu binary,
then request the list of implicit devices for each machine type
(libvirt runs each available qemu binary *once* the first time it's
requested, and caches all such capabilities information so that it
doesn't need to re-run qemu again and again). My limited understanding
of qemu's code is that qemu itself doesn't have a table of this
information as data, but instead has lines of code that are executed
to create it, thus making it impractical to provide the list of
devices for a machinetype without actually instantiating a machine of
that type. What's the feasibility of adding such a capability (and in
the process likely making the list of implicit devices in qemu itself
table/data driven rather than constructed with lines of code).
Machines are created by ad hoc board setup code. This code is generally
written to run once. And by once I mean one board gets initialized
once, and never destroyed. To support destruction, we'd have to find
out what needs to be cleaned up (allocated resources, global state,
...), and clean it up. For all of the boards. Even the barely
maintained ones.
Let's take a step back from the swamp we're in and examine the
swamp^H^H^H^H^Hmeadow we're trying to reach.
qdev used to be "declarative" in the sense that device models and their
properties are declared by data. Easy to introspect. The (far away)
goal was to extend this so that machine types become data, too.
QOM isn't declarative, it's fully dynamic, i.e. classes and properties
are created by code. I never quite understood why, but I'm sure there
are good reasons. I suspect the goal of having machine types as data
has been dropped, and the new plan is to create machines for
introspection. Assumes creating and destroying machines won't be a big
deal once they're fully QOMified.
More questions:
Alex replied to these, and I have nothing to add.
12:20 p.m.
New subject: [libvirt] [Qemu-devel] q35 machine type and libvirt.
On 02/06/2013 02:13 PM, Laine Stump wrote:
Now that qemu is getting the q35 machine type, libvirt needs to
support it.
In an attempt to make sure that libvirt actually does something useful
with qemu's new machine type, I'm revisiting this topic and trying to
get a more thorough understanding.
I've developed my own list of what I *think* are the reasons for wanting
a new machine type in qemu (and from that it should be more apparent
what libvirt needs to do with it), and am wondering how far off the mark
I am:
* Protection against obsolescence (PIIX (pc machine type) is 17 years old (?)
and some OSes may drop support for it).
* Passthrough of PCIe hardware? Is this really an issue? (e.g. the
VFs of a PCIe SRIOV network card can already be passed through as
standard PCI devices)
* Larger bus space? Is this solved even without the new machine type by simply
supporting the existing pci-bridge device and allowing multiple buses?
* Support for new emulated hardware. (what is important?)
Are any of these misguided thinking on my part? Are there other real advantages over using
the pc-* machine type?
As an adjunct to that, in some conversations people have mentioned the fact that the Q35
chipset is already out of production, and that supporting something more recent, like the
X58, might be better in the long run. Would the main advantage of that be supportability?
Or are there other concrete differences in a newer chipset that might by themselves make
it worth considering? (Conversely, would attempting to write the drivers for a newer
chipset just be busywork that only netted a more complex virtual machine but no useful
gains?)
So, based on the above (and whatever other gains are to be had from the new machine type)
what is needed/expected from libvirt? Here's my rough list:
* setup different default buses/controllers/devices based on machine
type (including possibility of using pcie.0 as the root)
* table of fixed locations for certain devices (if present) (again,
based on machine type)
* restrict certain *types* of devices to certain *types* of
slots? (I'm a bit fuzzy on this one, but this has to do with the
"root complex" vs. "endpoint" vs. "integrated endpoint"
that Alex
mentioned).
* support some new emulated chipset devices?
* Anything else specific required to passthrough pcie devices?
3:51 p.m.
New subject: [libvirt] [Qemu-devel] q35 machine type and libvirt.
On Wed, 2013-02-27 at 13:20 -0500, Laine Stump wrote:
On 02/06/2013 02:13 PM, Laine Stump wrote:
> Now that qemu is getting the q35 machine type, libvirt needs to support it.
In an attempt to make sure that libvirt actually does something useful
with qemu's new machine type, I'm revisiting this topic and trying to
get a more thorough understanding.
I've developed my own list of what I *think* are the reasons for wanting
a new machine type in qemu (and from that it should be more apparent
what libvirt needs to do with it), and am wondering how far off the mark
I am:
* Protection against obsolescence (PIIX (pc machine type) is 17 years old (?)
and some OSes may drop support for it).
* Passthrough of PCIe hardware? Is this really an issue? (e.g. the
VFs of a PCIe SRIOV network card can already be passed through as
standard PCI devices)
We can't expose PCIe extended capabilities without a PCI express
chipset. This includes error reporting and possibly use of advanced
features for iommu interaction and caching.
We also can't do any kind of IGD passthrough without a GMCH-like
chipset.
* Larger bus space? Is this solved even without the new machine
type by simply
supporting the existing pci-bridge device and allowing multiple buses?
* Support for new emulated hardware. (what is important?)
Emulated devices have the same extended capabilities limitation.
Are any of these misguided thinking on my part? Are there other real
advantages over using the pc-* machine type?
As an adjunct to that, in some conversations people have mentioned the fact that the Q35
chipset is already out of production, and that supporting something more recent, like the
X58, might be better in the long run. Would the main advantage of that be supportability?
Or are there other concrete differences in a newer chipset that might by themselves make
it worth considering? (Conversely, would attempting to write the drivers for a newer
chipset just be busywork that only netted a more complex virtual machine but no useful
gains?)
IMHO, X58 or newer Q77 chipset would only build on and swap out
components of something like Q35. Anything modern will have the same
basic layout; some concept of a PCIe root complex with PCIe-to-PCI
bridge(s) branching out to Legacy PCI buses and root ports, possibly
with PCIe switches, connecting to PCIe endpoints. All of that will be
the same for Q35/X58 or newer desktop chips.
So, based on the above (and whatever other gains are to be had from
the new machine type) what is needed/expected from libvirt? Here's my rough list:
* setup different default buses/controllers/devices based on machine
type (including possibility of using pcie.0 as the root)
* table of fixed locations for certain devices (if present) (again,
based on machine type)
* restrict certain *types* of devices to certain *types* of
slots? (I'm a bit fuzzy on this one, but this has to do with the
"root complex" vs. "endpoint" vs. "integrated
endpoint" that Alex
mentioned).
Right, somehow libvirt needs to know or qemu needs to tell it something
about the devices it's plugging in. If you were to grab your trust
10/100Mbps Legacy PCI ethernet card and try to plug it into a
motherboard you'd quickly notice that you can only plug it into certain
slots. This is the same problem. PCI device are attached to a legacy
PCI bus, which means it needs to be behind a PCIe-to-PCI bridge. Legacy
Endpoints and Endpoints are plugin PCIe cards, so they need to be
plugged in behind a PCIe switch or root port. Integrated Endpoints are
motherboard components, per the spec, they shouldn't even really be
hotplug-able. They attach directly to the root complex.
* support some new emulated chipset devices?
-M q35 + -device ioh3420 (root port) + -device i82801b11-bridge
(pcie-to-pci bridge)
* Anything else specific required to passthrough pcie devices?
I just sent out an RFC asking about VGA routing, it's possible libvirt
will need to understand some concept of a primary VGA device and specify
it somehow to QEMU.
It's also possible (probable) that we'll make assigned devices able to
mangle PCIe types to make it more flexible where they can be attached
(for instance you may not want to have your HD audio device exposed as a
root complex device if that disables hotplug, so we may mangle it to
look like a regular endpoint. Windows is picky about root complex
devices, so we may also mangle endpoint to integrated endpoint as well).
Also need to be aware that all PCIe "buses" except the root complex are
actually point-to-point links, so a root port connects to one PCIe
device (which may be multifunction). PCIe switches are needed to get
fan out to connect multiple devices, one device per downstream port. It
would be interesting how much we can abuse the topology in a virtual
system, but it's probably best not to confuse guests. Thanks,
Alex
6:46 p.m.
New subject: [libvirt] [Qemu-devel] q35 machine type and libvirt.
Alex Williamson <alex.williamson(a)redhat.com> writes:
On Wed, 2013-02-27 at 13:20 -0500, Laine Stump wrote:
> On 02/06/2013 02:13 PM, Laine Stump wrote:
> > Now that qemu is getting the q35 machine type, libvirt needs to support it.
>
> In an attempt to make sure that libvirt actually does something useful
> with qemu's new machine type, I'm revisiting this topic and trying to
> get a more thorough understanding.
>
> I've developed my own list of what I *think* are the reasons for wanting
> a new machine type in qemu (and from that it should be more apparent
> what libvirt needs to do with it), and am wondering how far off the mark
> I am:
>
Right, somehow libvirt needs to know or qemu needs to tell it something
about the devices it's plugging in. If you were to grab your trust
10/100Mbps Legacy PCI ethernet card and try to plug it into a
motherboard you'd quickly notice that you can only plug it into certain
slots. This is the same problem. PCI device are attached to a legacy
PCI bus, which means it needs to be behind a PCIe-to-PCI bridge. Legacy
Endpoints and Endpoints are plugin PCIe cards, so they need to be
plugged in behind a PCIe switch or root port. Integrated Endpoints are
motherboard components, per the spec, they shouldn't even really be
hotplug-able. They attach directly to the root complex.
We could do this with QOM. The chipset could have a set of link
properties for the integrated devices. For instance:
Q35Chipset
Link<E1000> integrated_nic;
Link<StdVGA> integrated_vga;
...
We should prevent PCI bus plugging for slots "owned" by integrated
devices. libvirt has a way to probe for links that it can add including
what types are allowed for it.
Regards,
Anthony Liguori
> * support some new emulated chipset devices?
-M q35 + -device ioh3420 (root port) + -device i82801b11-bridge
(pcie-to-pci bridge)
> * Anything else specific required to passthrough pcie devices?
I just sent out an RFC asking about VGA routing, it's possible libvirt
will need to understand some concept of a primary VGA device and specify
it somehow to QEMU.
It's also possible (probable) that we'll make assigned devices able to
mangle PCIe types to make it more flexible where they can be attached
(for instance you may not want to have your HD audio device exposed as a
root complex device if that disables hotplug, so we may mangle it to
look like a regular endpoint. Windows is picky about root complex
devices, so we may also mangle endpoint to integrated endpoint as well).
Also need to be aware that all PCIe "buses" except the root complex are
actually point-to-point links, so a root port connects to one PCIe
device (which may be multifunction). PCIe switches are needed to get
fan out to connect multiple devices, one device per downstream port. It
would be interesting how much we can abuse the topology in a virtual
system, but it's probably best not to confuse guests. Thanks,
Alex
4285
days inactive
4307
days old
8 comments
6 participants
participants (6)
-
Alex Williamson -
Anthony Liguori -
Daniel P. Berrange -
Laine Stump -
Markus Armbruster -
Michael S. Tsirkin