Happy new year! Nearly a year ago I reported an issue with the <hap> hypervisor feature on Xen [1] and am now seeing a similar issue with the <pae> feature. Setting the default value of pae changed between xend and libxl. When not specified, xend would enable pae for HVM domains. Clients such as xm and the old libvirt driver did not have to explicitly enable it. In libxl, the pae field within libxl_domain_build_info is initialized to 0. Clients must enable pae, and indeed xl will do so if pae=1 is not specified in the xl.cfg. The xend behavior prevents libvirt from disabling pae, whereas the libxl behvior causes a guest ABI change (config that worked with libvirt+xend doesn't with libvirt+libxl). The libxl behavior also forces management software (e.g. OpenStack nova) to add <pae> where it wasn't needed before. To solve this problem for <hap>, it was changed it to a tristate [2], allowing it to be turned off with explicit <hap state='off'/>, and on if not specified or <hap/> or <hap state='on'/>. Should <pae> (and the remaining hypervisor features that are not tristate) be converted to tristate similar to <hap>? Alternatively, I could simply set pae=1 for all HVM domains in the libxl driver. Like the old libvirt+xend behavior it couldn't be turned off, but I don't think there is a practical use-case to do so. At least no one has complained over all the years of libvirt+xend use.

[1] https://www.redhat.com/archives/libvir-list/2016-February/msg00197.html [2] https://www.redhat.com/archives/libvir-list/2016-March/msg00001.html

On Wed, Jan 11, 2017 at 10:49:48AM -0700, Jim Fehlig wrote: > On 01/06/2017 05:31 PM, Jim Fehlig wrote: > > Adding xen-devel for a question below... > >> Happy new year! >> >> Nearly a year ago I reported an issue with the <hap> hypervisor feature on Xen >> [1] and am now seeing a similar issue with the <pae> feature. Setting the >> default value of pae changed between xend and libxl. When not specified, xend >> would enable pae for HVM domains. Clients such as xm and the old libvirt driver >> did not have to explicitly enable it. In libxl, the pae field within >> libxl_domain_build_info is initialized to 0. Clients must enable pae, and indeed >> xl will do so if pae=1 is not specified in the xl.cfg. >> >> The xend behavior prevents libvirt from disabling pae, whereas the libxl behvior >> causes a guest ABI change (config that worked with libvirt+xend doesn't with >> libvirt+libxl). The libxl behavior also forces management software (e.g. >> OpenStack nova) to add <pae> where it wasn't needed before. >> >> To solve this problem for <hap>, it was changed it to a tristate [2], allowing >> it to be turned off with explicit <hap state='off'/>, and on if not specified or >> <hap/> or <hap state='on'/>. Should <pae> (and the remaining hypervisor features >> that are not tristate) be converted to tristate similar to <hap>? Alternatively, >> I could simply set pae=1 for all HVM domains in the libxl driver. Like the old >> libvirt+xend behavior it couldn't be turned off, but I don't think there is a >> practical use-case to do so. At least no one has complained over all the years >> of libvirt+xend use. > > Xen folks, what is your opinion of always enabling pae for HVM domains in > the libvirt libxl driver? Is there a need these days to disable it? > > Jan had mentioned that some old, buggy guest OS's (Win 9x) might need it > disabled, and perhaps some cases where it may be desirable to suppress a > guest OS entering 64-bit mode. But in practice do you think it is necessary > to expose this knob to users? If the guest arch is declared as x86_64, then you should unconditionally force 'pae' to be present, otherwise you'd be giving the guest an i686 environment.

The app should request arch == i686 explicitly if they want an i686 environment. Allowing pae to be toggled for i686 guests is ok, since that indicates whethre the i686 CPU can address > 2GB of RAM.

On 11/01/17 17:49, Jim Fehlig wrote: > On 01/06/2017 05:31 PM, Jim Fehlig wrote: > > Adding xen-devel for a question below... > >> Happy new year! >> >> Nearly a year ago I reported an issue with the <hap> hypervisor >> feature on Xen >> [1] and am now seeing a similar issue with the <pae> feature. Setting >> the >> default value of pae changed between xend and libxl. When not >> specified, xend >> would enable pae for HVM domains. Clients such as xm and the old >> libvirt driver >> did not have to explicitly enable it. In libxl, the pae field within >> libxl_domain_build_info is initialized to 0. Clients must enable pae, >> and indeed >> xl will do so if pae=1 is not specified in the xl.cfg. >> >> The xend behavior prevents libvirt from disabling pae, whereas the >> libxl behvior >> causes a guest ABI change (config that worked with libvirt+xend >> doesn't with >> libvirt+libxl). The libxl behavior also forces management software (e.g. >> OpenStack nova) to add <pae> where it wasn't needed before. >> >> To solve this problem for <hap>, it was changed it to a tristate [2], >> allowing >> it to be turned off with explicit <hap state='off'/>, and on if not >> specified or >> <hap/> or <hap state='on'/>. Should <pae> (and the remaining >> hypervisor features >> that are not tristate) be converted to tristate similar to <hap>? >> Alternatively, >> I could simply set pae=1 for all HVM domains in the libxl driver. >> Like the old >> libvirt+xend behavior it couldn't be turned off, but I don't think >> there is a >> practical use-case to do so. At least no one has complained over all >> the years >> of libvirt+xend use. > > Xen folks, what is your opinion of always enabling pae for HVM domains > in the libvirt libxl driver? Is there a need these days to disable it? > > Jan had mentioned that some old, buggy guest OS's (Win 9x) might need > it disabled, and perhaps some cases where it may be desirable to > suppress a guest OS entering 64-bit mode. But in practice do you think > it is necessary to expose this knob to users? ISTR the main use of this knob being to cause 32bit versions of windows to avoid using PAE paging, making them more efficient to shadow. Now that 64bit and EPT/NPT are fairly ubiquitous, there should be no reason to turn it off.

Can people still play with the CPUID policy if they really need to disable it?

It is unfortunate that this option was ever exposed via a non-CPUID mechanism, but I am trying to clean this up at the hypervisor interface to ensure that the *only* way to alter details like this is via the appropriate interface.