[libvirt] [BUG] Xen x86_64 needs PAE?
Hello, I tried to install "Microsoft Windows 2008 Server R2" on an Amd64 host using Xen-3.4.3. The installation aborted with the following error message:
Windows Boot Mager ... File: \windows\system32\boot\winload.exe Status: 0xc000035a Info: Attempting to load a 64-bit application, however this CPU is not compatible with 64-bit mode.
For testing I converted the libvirt-XML file to an Xen-Xm file using "virsh domxml-to-native" and started that file with "xm create", which worked. Comparing the output of "xm list -a" of the broken libvirt- and the working xm-configuration I found several differences, especially with "pae": Using "xm create" pae is set to 1, while with libvirt pae is set to 0. After adding the /domain/featues/pae node to libvirts XML domain file, Windows installed fine. Looking at "virsh capabilities" I get the following (abbreviated) output:
<capabilities> <guest> <os_type>hvm</os_type> <arch name='i686'> <wordsize>32</wordsize> ... </arch> <features> <pae/> <nonpae/> <acpi default='on' toggle='yes'/> <apic default='on' toggle='yes'/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='x86_64'> <wordsize>64</wordsize> ... </arch> <features> <acpi default='on' toggle='yes'/> <apic default='on' toggle='yes'/> </features> </guest> </capabilities>
As you can see, neither "<pae/>" nor "<nopae/>" is listed for for the 64 bit Xen-Fv domain type, which makes it hard to automatically create a correct _and_ _working_ XML file for Xen-Fv-64 bit domains Looking at "/sys/hypervisor/properties/capabilities" I find the following capabilities reported by Xen
xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
There's also no "hvm-3.0-x86_64p" which makes me wonder, if "<pae/>" is the default for Xen-64 bit domains. If that is true, should libvirt add "<pae/"> by default to the capabilities report, or should libvirt silently pass "(pae 1)" when pushing the XML domain description to Xend? Lookign at <file:xen-3.4.3/tools/python/xen/xm/create.py>:210 I see that "pae" is enabled by default there, and thus gets enabled for each domain using "xm create":
gopts.var('pae', val='PAE', fn=set_int, default=1, use="Disable or enable PAE of HVM domain.")
Any feedback is appreciated. BYtE Philipp -- Philipp Hahn Open Source Software Engineer hahn@univention.de Univention GmbH Linux for Your Business fon: +49 421 22 232- 0 Mary-Somerville-Str.1 28359 Bremen fax: +49 421 22 232-99 http://www.univention.de
On Mon, Aug 09, 2010 at 02:20:42PM +0200, Philipp Hahn wrote:
Hello,
I tried to install "Microsoft Windows 2008 Server R2" on an Amd64 host using Xen-3.4.3. The installation aborted with the following error message:
Windows Boot Mager ... File: \windows\system32\boot\winload.exe Status: 0xc000035a Info: Attempting to load a 64-bit application, however this CPU is not compatible with 64-bit mode.
For testing I converted the libvirt-XML file to an Xen-Xm file using "virsh domxml-to-native" and started that file with "xm create", which worked. Comparing the output of "xm list -a" of the broken libvirt- and the working xm-configuration I found several differences, especially with "pae": Using "xm create" pae is set to 1, while with libvirt pae is set to 0. After adding the /domain/featues/pae node to libvirts XML domain file, Windows installed fine.
If you can provide the 'xm' config file, the libvirt XML and the output of 'xm list --list $GUESTNAME' when the guest is succesfully running we can add a test case to validate this PAE setup & fix the xen driver.
Looking at "virsh capabilities" I get the following (abbreviated) output:
<capabilities> <guest> <os_type>hvm</os_type> <arch name='i686'> <wordsize>32</wordsize> ... </arch> <features> <pae/> <nonpae/> <acpi default='on' toggle='yes'/> <apic default='on' toggle='yes'/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='x86_64'> <wordsize>64</wordsize> ... </arch> <features> <acpi default='on' toggle='yes'/> <apic default='on' toggle='yes'/> </features> </guest> </capabilities>
As you can see, neither "<pae/>" nor "<nopae/>" is listed for for the 64 bit Xen-Fv domain type, which makes it hard to automatically create a correct _and_ _working_ XML file for Xen-Fv-64 bit domains
Looking at "/sys/hypervisor/properties/capabilities" I find the following capabilities reported by Xen
xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
There's also no "hvm-3.0-x86_64p" which makes me wonder, if "<pae/>" is the default for Xen-64 bit domains.
If that is true, should libvirt add "<pae/"> by default to the capabilities report, or should libvirt silently pass "(pae 1)" when pushing the XML domain description to Xend? Lookign at
In 32-bit Xen, the pae flag indicates whether this is a 32bit PAE or non-PAE guest. In 64-bit Xen, the pae flag indicates whether the CPU exposed supports long mode. So if you don't set the PAE flag on 64-bit Xen, you get a 32-bit PAE guest. If you do set the PAE flag, you get a 64-bit guest. Yes, you read that right, not setting PAE, gets you a PAE enabled guest just with 32 instead of 64 bits :-) In terms of the domain XML design, the choice between 32 and 64 bits is intended to be done based on the arch field: <os> <type arch='x86_64'... ... </os> The pae flag in domain XML should be reflecting whether pae is enabled in the guest. The capabilities XML should be reporting whether the guest ABI includes the PAE flag or not. So I think the 32bit capabilities shouldn't have included <nonpae/> there, and perhaps 64-bit should have had <pae/>. Ultimately though I think our mistake is in passing the <pae> flag from the domain XML, straight through to the hypervisor. I'm rather wary of changing this, because of the risk of regressions, but if you can supply the 3 files I asked for above, we can investigate whether its feasible Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Hello, Am Dienstag 24 August 2010 14:12:24 schrieb Daniel P. Berrange:
If you can provide the 'xm' config file, the libvirt XML and the output of 'xm list --list $GUESTNAME' when the guest is succesfully running we can add a test case to validate this PAE setup & fix the xen driver.
I attached 8 files: 1. xen-fv-w2k8.broken.xml is my initial file, which was used with 'virsh define' + 'virsh start'. 2. xen-fv-w2k8.broken.xm was created using "virsh domxml-to-native xen-xm" 3. xen-fv-w2k8.broken.dump was created using "xm list --long Windows2008r2_64" 4. xen-fv-w2k8.fixed.xml with just '<pae/>' added, 'virsh define' + 'virsh start'. 5.-6. xen-fv-w2k8.fixed.{xm,dump} analog to above. 7. xen-fv-w2k8.clean.xm is without the pae=[01] line, so 'xm create' uses it's default. 8. xen-fv-w2k8.clean.dump of 'xm list --long' after 'xm create' 501 root@xen1=xen64:~ 0# xm info host : xen1 release : 2.6.32-ucs11-xen-amd64 version : #1 SMP Sat Jul 31 07:49:13 UTC 2010 machine : x86_64 hw_caps : 078bf3ff:ebd3fbff:00000000:00000010:00002001:0000001d:0000001d:00000000 virt_caps : hvm xen_major : 3 xen_minor : 4 xen_extra : .3 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
In 64-bit Xen, the pae flag indicates whether the CPU exposed supports long mode. So if you don't set the PAE flag on 64-bit Xen, you get a 32-bit PAE guest. If you do set the PAE flag, you get a 64-bit guest. Yes, you read that right, not setting PAE, gets you a PAE enabled guest just with 32 instead of 64 bits :-)
In terms of the domain XML design, the choice between 32 and 64 bits is intended to be done based on the arch field:
<os> <type arch='x86_64'... ... </os>
That is what I expected as well: neither for Windows 2k8 nor for Linux would I get a 64 bit guest. Only after adding the <pae/> feature it worked. Perhaps related: Setting "arch=x86_64" for Xen domains never made a difference for me: On the next "virsh dumpxml" that attribute was gone again making it impossible to distinguish between 32 and 64 bit Xen domains.
The pae flag in domain XML should be reflecting whether pae is enabled in the guest.
If I understand pae right, x86_64 used a page-table-layout very similar to x86_32+pae, which makes doing the Virtual Memory mapping between host and guests easy. For x86_32 without pae you'd need to do some heavy reshuffling, because the page tables would be totally different. I noticed the pae problem during my testing: guests started via "xm create/start" ran fast, while guestss started via "virsh start" were very slow. As soon as one non-pae-guest was running, the whole system got unresponsive. It seemed to be related to the memory size, since running a guest with 128M or 512M made a huge difference.
The capabilities XML should be reporting whether the guest ABI includes the PAE flag or not. So I think the 32bit capabilities shouldn't have included <nonpae/> there, and perhaps 64-bit should have had <pae/>. Ultimately though I think our mistake is in passing the <pae> flag from the domain XML, straight through to the hypervisor. I'm rather wary of changing this, because of the risk of regressions, but if you can supply the 3 files I asked for above, we can investigate whether its feasible
The missing <pae/> is translated to pae=0, which overwrites Xens default. Because of that my code currently tests for xen on x86_64 and than always adds <pae/> to make it work. My expectation would be that the capabilities XML should thus report <pae/>, if an application is supposed to add the <pae/> feature to get an 64 bit xen-fv domain. Sincerely Philipp -- Philipp Hahn Open Source Software Engineer hahn@univention.de Univention GmbH Linux for Your Business fon: +49 421 22 232- 0 Mary-Somerville-Str.1 28359 Bremen fax: +49 421 22 232-99 http://www.univention.de **** Besuchen Sie unsere Praxis-Vorträge zu UCS@School auf der FIT-ÖV am 23.09.2010 im Haus der Wissenschaft in Bremen ****
participants (2)
-
Daniel P. Berrange -
Philipp Hahn