[Libvir] big-endian support for libvirt - introduce GUEST_HANDLE infrastructure ?
Hello everyone, we currently try to use/port libvirt to run on xenppc. While doing that I found a endianess issue. First a simple example: Sample code: #include <stdio.h> union u { int i; long long l; }; int main(void) { union u u; u.l = 0; u.i = ~0; printf("%016llx\n", u.l); return 0; } Little-endian output: 00000000ffffffff Big-endian output: ffffffff00000000 Now look at the padding in e.g. xen_v2s3_getdomaininfolistop: it doesn't work on big-endian systems. This problem is why the GUEST_HANDLE infrastructure is present in Xen. To work on big-endian systems, libvirt will need this or a similar mechanism. --- Current Questions --- We could now add several ugly ifdefs to libvirt code to differentiate powerpc from the others and solve the issue described above, but I think thats not a good solution. The GUEST_HANDLE mechanism would provide an architecture abstraction and is already implemented/working in libxc. The question is, shouldn't libvirt actually use the GUEST_HANDLE mechanism like libxc does (at least for the _v2d5_ structures) or are there big arguments against it? I would create a patch to add the GUEST_HANDLE stuff in libvirt if there is nothing against it, but I will need some help to test it on x86/x86_64/ia64 since I have no such machines here. If there is a major reason against the GUEST_HANDLE code, are there other suggestions/preferences how to solve this? Below is the technical background in a more verbose way for anyone interested --- Code Affected --- More or less everything in libvirt that would use GUEST_HANDLE in libxc: I saw the effect on a bad pointer passed down by the structure I use for my example below, but in the end it may affect all the stuff that uses the *GUEST_HANDLE* stuff in libxc which are e.g. uchar,ulong,... here an example: libxc: uint64_aligned_t max_pages; // <- arch dependent mapping of uint64_aligned_t libvirt: #define ALIGN_64 __attribute__((aligned(8))) uint64_t max_pages ALIGN_64; // <- this align is fix on all architectures --- The Issue in libvirt in one example --- XEN_GUEST_HANDLE usage in libxc: - XEN_GUEST_HANDLE is defined arch specific giving the arch the posibility to pad/align as needed. See xen/tools/libxc/xen/arch-powerpc.h for example. - xen/tools use set_xen_guest_handle and get_xen_guest_handle to set/access these handles and let the arch the posibility to handle any specific needs. Example: struct xen_sysctl_getdomaininfolist { domid_t first_domain; uint32_t max_domains; XEN_GUEST_HANDLE_64(xen_domctl_getdomaininfo_t) buffer; uint32_t num_domains; }; The arch defined XEN_GUEST_HANDLE_64 leads to the following type for buffer on powerpc (see the initial padding): typedef struct { \ int __pad[(sizeof (long long) - sizeof (void *)) / sizeof (int)]; \ xen_domctl_getdomaininfo_t *p; } __attribute__((__aligned__(8))) __guest_handle_xen_domctl_getdomaininfo_t In this scenario a 32bit pointer for *p with 0xFFFFFFFF will be stored as 0xPADPADPAFFFFFFFF and read in a valid way by the 64bit host that receives this syscall. XEN_GUEST_HANDLE usage in libvirt: - In my example where XEN_GUEST_HANDLE is in libxc libvirt currently uses a union based structure like the following: struct xen_v2s3_getdomaininfolistop { domid_t first_domain; uint32_t max_domains; union { struct xen_v2d5_getdomaininfo *v; uint64_t pad ALIGN_64; } buffer; uint32_t num_domains; }; typedef struct xen_v2s3_getdomaininfolistop xen_v2s3_getdomaininfolistop; The union used here to add the 64 bit padding does not work on big endian powerpc because reading this two half words in 64bit receiver gives 0xFFFFFFFF00000000 (The two half words are stored in the wrong order) while on ia64 it is read as the the valid 0x00000000FFFFFFFF. P.S. Thanks Hollis for the simple example code -- Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, Open Virtualization +49 7031/16-3385 Ehrhardt@linux.vnet.ibm.com Ehrhardt@de.ibm.com IBM Deutschland Entwicklung GmbH Vorsitzender des Aufsichtsrats: Johann Weihen Geschäftsführung: Herbert Kircher Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
Christian Ehrhardt wrote:
Now look at the padding in e.g. xen_v2s3_getdomaininfolistop: it doesn't work on big-endian systems. This problem is why the GUEST_HANDLE infrastructure is present in Xen. To work on big-endian systems, libvirt will need this or a similar mechanism.
Understood.
--- Current Questions --- We could now add several ugly ifdefs to libvirt code to differentiate powerpc from the others and solve the issue described above, but I think thats not a good solution. The GUEST_HANDLE mechanism would provide an architecture abstraction and is already implemented/working in libxc. The question is, shouldn't libvirt actually use the GUEST_HANDLE mechanism like libxc does (at least for the _v2d5_ structures) or are there big arguments against it? I would create a patch to add the GUEST_HANDLE stuff in libvirt if there is nothing against it, but I will need some help to test it on x86/x86_64/ia64 since I have no such machines here. If there is a major reason against the GUEST_HANDLE code, are there other suggestions/preferences how to solve this?
I had a look at the GUEST_HANDLE code in libxc (specifically arch-powerpc.h vs arch-x86_{32,64}.h) and it looks as if something like this should go into libvirt. I would just be careful about copying the code verbatim because libvirt & libxc are under slightly different licenses. As for testing: we have no 64 bit PPC machines at all available for us to use that I'm aware of. We have plenty of i386 & x86-64, and one or two IA64 machines. Thanks for your analysis. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903
On Fri, Jul 06, 2007 at 11:06:54AM +0200, Christian Ehrhardt wrote:
Hello everyone, we currently try to use/port libvirt to run on xenppc.
Hello, cool :-)
While doing that I found a endianess issue.
less cool <grin/> [...]
Now look at the padding in e.g. xen_v2s3_getdomaininfolistop: it doesn't work on big-endian systems. This problem is why the GUEST_HANDLE infrastructure is present in Xen. To work on big-endian systems, libvirt will need this or a similar mechanism.
--- Current Questions --- We could now add several ugly ifdefs to libvirt code to differentiate powerpc from the others and solve the issue described above, but I think thats not a good solution.
Really ? I did a quick scan in xen_internal.c , that's the only place where the problem might be raised, and found only 3 structure where the problem was likely to occur: xen_v2s3_getdomaininfolistop, xen_v2d5_cpumap, and xen_v2_setschedinfo, the two first ones clearly the last one is less clear. Detecting big-endian vs. little can easilly be done in configure or inherited from a system include and a couple of struct xen_v2d5_cpumap { #ifdef LITTLE_ENDIAN union { uint8_t *v; uint64_t pad ALIGN_64; } bitmap; #else struct { uint8_t *pad; uint8_t *v; } bitmap; #endif uint32_t nr_cpus; }; Might be easier to understand than the GUEST_HANDLE complex macros.
The GUEST_HANDLE mechanism would provide an architecture abstraction and is already implemented/working in libxc. The question is, shouldn't libvirt actually use the GUEST_HANDLE mechanism like libxc does (at least for the _v2d5_ structures) or are there big arguments against it?
I'm not sure it helps on code maintainance, since it would affect only a couple of places in libvirt. Also we definitely cannot copy code from libxc due to licence incompatibilities, avoiding reusing the exact same mechanism makes it even clearer it's a rewrite. [...]
--- Code Affected --- More or less everything in libvirt that would use GUEST_HANDLE in libxc: I saw the effect on a bad pointer passed down by the structure I use for my example below, but in the end it may affect all the stuff that uses the *GUEST_HANDLE* stuff in libxc which are e.g. uchar,ulong,... here an example:
How many places in libvirt are really affected ? I count 3, we use a very small subset of the hypercalls, but I may be wrong. If there is really more places then maybe a macro based mechanism is better maintainance wise but in any case it would have to be rewritten from scratch to avoid nervosity from our Xensource friends. Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
Really ? I did a quick scan in xen_internal.c , that's the only place where the problem might be raised, and found only 3 structure where the problem was likely to occur: xen_v2s3_getdomaininfolistop, xen_v2d5_cpumap, and xen_v2_setschedinfo, the two first ones clearly the last one is less clear. Detecting big-endian vs. little can easilly be done in configure or inherited from a system include and a couple of
Thats right - after looking around more in detail I also see not more than the listed places at the moment. I did not expect that it is only such a small subset, but it is good that way in reference to the current issue ;-)
struct xen_v2d5_cpumap { #ifdef LITTLE_ENDIAN union { uint8_t *v; uint64_t pad ALIGN_64; } bitmap; #else struct { uint8_t *pad; uint8_t *v; } bitmap; #endif uint32_t nr_cpus; };
Might be easier to understand than the GUEST_HANDLE complex macros.
[...]
How many places in libvirt are really affected ? I count 3, we use a very small subset of the hypercalls, but I may be wrong. If there is really more places then maybe a macro based mechanism is better maintainance wise but in any case it would have to be rewritten from scratch to avoid nervosity from our Xensource friends.
You are right, for the moment it is really sufficient to fix xen_v2s3_getdomaininfolistop and xen_v2d5_cpumap. I'll do something that way dependent on configure&ifdef and send a patch (low prio since we have a workaround in the kernel)
Daniel
BTW here is a short status which libvirt functions are working on xenppc as seen on my first sniff test: working *monitor load *remove network interface *remove hard disk *access serial console *define existing xen domains from/to xml file *run/shutdown domains partially working *add network device (work but cause a libvirt trace&hang afterwards) *add hard disk (work but cause a libvirt trace&hang afterwards) not working *sometimes xend hangs and only /etc/init.d/xend restart helps, there are related tracebacks in xend.log *change number of cpu's *create virtual network -- Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, Open Virtualization +49 7031/16-3385 Ehrhardt@linux.vnet.ibm.com Ehrhardt@de.ibm.com IBM Deutschland Entwicklung GmbH Vorsitzender des Aufsichtsrats: Johann Weihen Geschäftsführung: Herbert Kircher Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On Tue, Jul 10, 2007 at 02:02:19PM +0200, Christian Ehrhardt wrote:
Really ? I did a quick scan in xen_internal.c , that's the only place where the problem might be raised, and found only 3 structure where the problem was likely to occur: xen_v2s3_getdomaininfolistop, xen_v2d5_cpumap, and xen_v2_setschedinfo, the two first ones clearly the last one is less clear. Detecting big-endian vs. little can easilly be done in configure or inherited from a system include and a couple of
Thats right - after looking around more in detail I also see not more than the listed places at the moment. I did not expect that it is only such a small subset, but it is good that way in reference to the current issue ;-)
yeah, the scope of the troubles is limited. Future needs to use the hypercall could be for example for more fine grained monitoring/profiling, but I don't expect to add much in a near future. Next step in term of functionality is clearly migration but that will go though xend, not the hypervisor.
struct xen_v2d5_cpumap { #ifdef LITTLE_ENDIAN union { uint8_t *v; uint64_t pad ALIGN_64; } bitmap; #else struct { uint8_t *pad; uint8_t *v; } bitmap; #endif uint32_t nr_cpus; };
Might be easier to understand than the GUEST_HANDLE complex macros.
[...]
How many places in libvirt are really affected ? I count 3, we use a very small subset of the hypercalls, but I may be wrong. If there is really more places then maybe a macro based mechanism is better maintainance wise but in any case it would have to be rewritten from scratch to avoid nervosity from our Xensource friends.
You are right, for the moment it is really sufficient to fix xen_v2s3_getdomaininfolistop and xen_v2d5_cpumap.
Good :-)
I'll do something that way dependent on configure&ifdef and send a patch (low prio since we have a workaround in the kernel)
That would be perfect ! Maybe we don't even need to hook in configure I'm sure endianness info can come from standard headers, then combined with a processor check that should be sufficient I guess.
BTW here is a short status which libvirt functions are working on xenppc as seen on my first sniff test: working *monitor load *remove network interface *remove hard disk *access serial console *define existing xen domains from/to xml file *run/shutdown domains
okay
partially working *add network device (work but cause a libvirt trace&hang afterwards) *add hard disk (work but cause a libvirt trace&hang afterwards)
weird, we are actually calling xend for this, that should not be platform specific, could be an hidden generic bug
not working *sometimes xend hangs and only /etc/init.d/xend restart helps, there are related tracebacks in xend.log
that's libvirt related ?
*change number of cpu's
hum, we do lot of bitfield operations there, there is certainly some cleanups needed, so that's not surprizing
*create virtual network
could be the calls to brctl or the iptable interfaces, some debugging certainly needed. Thanks a lot for the feedback, and don't hesitate to ask ! Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
That would be perfect ! Maybe we don't even need to hook in configure I'm sure endianness info can come from standard headers, then combined with a processor check that should be sufficient I guess.
Without introducing all the guest handle infrastructure and by just fixing the known xen_v2s3_getdomaininfolistop and xen_v2d5_cpumap the patch became as nice and small as a patch should be ;-) The Libvirt padding is now handled by using gcc's __BIG_ENDIAN__, no configure/header/... needed that way. I think we can assume that at least when libvirt is compiled for ppc gcc is used right? I used one (1) really generic line from our ppc libxc code, this should be no licencing issue (LGPL vs. GPL). Tell me if someone think otherwise and I'll try to change it a bit. Since it is no longer a macro we could pre-calculate the sizes anyway, but the way it is now says clearly "64bit size - used size" for the padding and I like that kind of readability. -- Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, Open Virtualization +49 7031/16-3385 Ehrhardt@linux.vnet.ibm.com Ehrhardt@de.ibm.com IBM Deutschland Entwicklung GmbH Vorsitzender des Aufsichtsrats: Johann Weihen Geschäftsführung: Herbert Kircher Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On Wed, Jul 11, 2007 at 02:11:38PM +0200, Christian Ehrhardt wrote:
That would be perfect ! Maybe we don't even need to hook in configure I'm sure endianness info can come from standard headers, then combined with a processor check that should be sufficient I guess.
Without introducing all the guest handle infrastructure and by just fixing the known xen_v2s3_getdomaininfolistop and xen_v2d5_cpumap the patch became as nice and small as a patch should be ;-)
yup, see it suddenly becomes quite small :-) maybe a tad bit too small though
The Libvirt padding is now handled by using gcc's __BIG_ENDIAN__, no configure/header/... needed that way. I think we can assume that at least when libvirt is compiled for ppc gcc is used right?
yes the only potential problem would be with other architectures where __BIG_ENDIAN__ is defined and where the relative size of pointers and long would be different.
I used one (1) really generic line from our ppc libxc code, this should be no licencing issue (LGPL vs. GPL). Tell me if someone think otherwise and I'll try to change it a bit.
No I guess it's generic enough :-)
Since it is no longer a macro we could pre-calculate the sizes anyway, but the way it is now says clearly "64bit size - used size" for the padding and I like that kind of readability.
Yes this makes for very long line, but it's really not a concern. I'm tempted to apply that patch after a day of grace delay to give people a chance to voice in ! thanks a lot ! Does this fix all the libvirt proper platform issues (i.e. independantly of possible xen specific ones) ? Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
Daniel Veillard wrote:
On Wed, Jul 11, 2007 at 02:11:38PM +0200, Christian Ehrhardt wrote: [...]
yes the only potential problem would be with other architectures where __BIG_ENDIAN__ is defined and where the relative size of pointers and long would be different.
We can change it to something more generic the moment we recognize other affected architectures. Or in a free minute ;-) [...]
Since it is no longer a macro we could pre-calculate the sizes anyway, but the way it is now says clearly "64bit size - used size" for the padding and I like that kind of readability.
Yes this makes for very long line, but it's really not a concern. I'm tempted to apply that patch after a day of grace delay to give people a chance to voice in !
thanks a lot ! Does this fix all the libvirt proper platform issues (i.e. independantly of possible xen specific ones) ?
Yes it fixes them as far as they are currently known to me. As I wrote before I had already a temporary workaround in the kernel, so the libvirt on xenppc status I posted a few mails before is valid for the patched libvirt&kernel scenario.
Daniel
-- Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, Open Virtualization +49 7031/16-3385 Ehrhardt@linux.vnet.ibm.com Ehrhardt@de.ibm.com IBM Deutschland Entwicklung GmbH Vorsitzender des Aufsichtsrats: Johann Weihen Geschäftsführung: Herbert Kircher Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On Wed, 2007-07-11 at 15:48 +0200, Christian Ehrhardt wrote:
thanks a lot ! Does this fix all the libvirt proper platform issues (i.e. independantly of possible xen specific ones) ?
Yes it fixes them as far as they are currently known to me. As I wrote before I had already a temporary workaround in the kernel, so the libvirt on xenppc status I posted a few mails before is valid for the patched libvirt&kernel scenario.
Christian, I don't remember seeing your kernel patch; could you repost it? If we are munging something in the kernel because userspace calls us with improper parameters, I think it goes without saying that we should fix userspace... -- Hollis Blanchard IBM Linux Technology Center
Hollis Blanchard wrote:
On Wed, 2007-07-11 at 15:48 +0200, Christian Ehrhardt wrote:
thanks a lot ! Does this fix all the libvirt proper platform issues (i.e. independantly of possible xen specific ones) ?
Yes it fixes them as far as they are currently known to me. As I wrote before I had already a temporary workaround in the kernel, so the libvirt on xenppc status I posted a few mails before is valid for the patched libvirt&kernel scenario.
Christian, I don't remember seeing your kernel patch; could you repost it?
It was here on xenppc list one minute after the patch to libvirt on this list ;-) http://www.mail-archive.com/xen-ppc-devel@lists.xensource.com/msg01982.html http://www.mail-archive.com/xen-ppc-devel@lists.xensource.com/msg01983.html
If we are munging something in the kernel because userspace calls us with improper parameters, I think it goes without saying that we should fix userspace...
The mugging was the temporary workaround I posted on xenppc last week which allowed me to continue libvirt testing until the libvirt patch (this thread) is finished and applied. The last patch on xenppc (two links above) now replaces the mugging code with sanity check to get a useful info in the case an old libvirt code level (or something else) issues a corrupted address. -- Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, Open Virtualization +49 7031/16-3385 Ehrhardt@linux.vnet.ibm.com Ehrhardt@de.ibm.com IBM Deutschland Entwicklung GmbH Vorsitzender des Aufsichtsrats: Johann Weihen Geschäftsführung: Herbert Kircher Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
Daniel Veillard wrote:
The Libvirt padding is now handled by using gcc's __BIG_ENDIAN__, no configure/header/... needed that way. I think we can assume that at least when libvirt is compiled for ppc gcc is used right?
yes the only potential problem would be with other architectures where __BIG_ENDIAN__ is defined and where the relative size of pointers and long would be different.
I guess the only potential problem would be on IA64 in big-endian mode. Does anyone use IA64 in this mode? I saw that Xen supports it ... As for the patch in general, since it doesn't affect any other platforms that libvirt currently supports (all little endian), I would apply it. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903
On Wed, Jul 11, 2007 at 08:50:23AM -0400, Daniel Veillard wrote:
On Wed, Jul 11, 2007 at 02:11:38PM +0200, Christian Ehrhardt wrote:
Without introducing all the guest handle infrastructure and by just fixing the known xen_v2s3_getdomaininfolistop and xen_v2d5_cpumap the patch became as nice and small as a patch should be ;-) [...] I'm tempted to apply that patch after a day of grace delay to give people a chance to voice in !
Okidoc, applied and commited to CVS, thanks a lot ! Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Fri, Jul 06, 2007 at 11:06:54AM +0200, Christian Ehrhardt wrote:
Hello everyone, we currently try to use/port libvirt to run on xenppc. While doing that I found a endianess issue.
First a simple example:
Sample code: #include <stdio.h>
union u { int i; long long l; };
int main(void) { union u u; u.l = 0; u.i = ~0; printf("%016llx\n", u.l);
return 0; }
Little-endian output: 00000000ffffffff Big-endian output: ffffffff00000000
Now look at the padding in e.g. xen_v2s3_getdomaininfolistop: it doesn't work on big-endian systems. This problem is why the GUEST_HANDLE infrastructure is present in Xen. To work on big-endian systems, libvirt will need this or a similar mechanism.
--- Current Questions --- We could now add several ugly ifdefs to libvirt code to differentiate powerpc from the others and solve the issue described above, but I think thats not a good solution. The GUEST_HANDLE mechanism would provide an architecture abstraction and is already implemented/working in libxc.a
libxc is GPL, while libvirt is LGPL.
The question is, shouldn't libvirt actually use the GUEST_HANDLE mechanism like libxc does (at least for the _v2d5_ structures) or are there big arguments against it?
We can't use the structures directly because libvirt needs to compile in support for *all* hypervisor ABIs so it can automatically use the correct ABI at runtime. We don't want a build of libvirt to be tied to the HV it was built against because that creates serious pain in upgrades.
More or less everything in libvirt that would use GUEST_HANDLE in libxc: I saw the effect on a bad pointer passed down by the structure I use for my example below, but in the end it may affect all the stuff that uses the *GUEST_HANDLE* stuff in libxc which are e.g. uchar,ulong,... here an example: libxc: uint64_aligned_t max_pages; // <- arch dependent mapping of uint64_aligned_t libvirt: #define ALIGN_64 __attribute__((aligned(8))) uint64_t max_pages ALIGN_64; // <- this align is fix on all architectures
Would it be sufficient to add #ifndef PPC or some kind of conditional around the '#define ALIGN_64' definitions ?
XEN_GUEST_HANDLE usage in libvirt: - In my example where XEN_GUEST_HANDLE is in libxc libvirt currently uses a union based structure like the following: struct xen_v2s3_getdomaininfolistop { domid_t first_domain; uint32_t max_domains; union { struct xen_v2d5_getdomaininfo *v; uint64_t pad ALIGN_64; } buffer; uint32_t num_domains; }; typedef struct xen_v2s3_getdomaininfolistop xen_v2s3_getdomaininfolistop;
The union used here to add the 64 bit padding does not work on big endian powerpc because reading this two half words in 64bit receiver gives 0xFFFFFFFF00000000 (The two half words are stored in the wrong order) while on ia64 it is read as the the valid 0x00000000FFFFFFFF.
I'm fine with any suggestions for changes to the structs above to make it work correctly on big-endian, provided they don't involve copying code from libxc Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
participants (5)
-
Christian Ehrhardt -
Daniel P. Berrange -
Daniel Veillard -
Hollis Blanchard -
Richard W.M. Jones