[libvirt] Libvirt PHP binding

Hi, I am trying to develop PHP binding for Libvirt API. As discussed previously on the list the best way is to create a ZEND extension. There is a great tutorial on the net ( http://devzone.zend.com/tag/Extension ) and PHP provides a lot of macros and functions to make extension writing easier. I've implemented functions for connecting, listing domains, getting info about domains and node, dumping domain XML, some basic domain life-cycle actions and very basic error reporting. Before moving to other functions I need to solve a few issues :-) The biggest issue is that PHP used from webserver can (and usually does) run multithread. One thread can process more than one request and the threads can run in parallel. PHP/ZEND provides "tools" to solve this. It provides "request-global" variables, resources (containers for objects like file handles or connection pointers) and much more. In single threaded environment (like calling from command line) all these fallback to usual behavior. In addition ZEND has its own memory manager to be able to better manage the memory on per request basis (functions are called emalloc(), efree(),...). One can still use malloc() and free() but this is bypassing the ZEND memory manager. Free()ing emalloced memory or vice versa leads to crash. So one thing is that every memory allocated by libvirt (using malloc()) I need to copy to memory allocated using emalloc(). Also every memory, that would be free()ed by libvirt I need to malloc() and copy from emmaloc()ed memory. The first one is done by simple macro: #define RECREATE_STRING_WITH_E(str_out, str_in) \ str_out = estrndup(str_in, strlen(str_in)); \ free(str_in); The second case I encountered only on authentication callback and I use simple strndup() to copy it for libvirt. When running from command line everything seems to work fine. This is of course singlethreaded and potential resource leaks need not to cause a problem. However when running from webserver, the result is much more interesting. When connecting to qemu:///system (via local socket) using <? libvirt_connect($uri,true); ?> I can crash the libvirt daemon after cca 10 page reloads - sometimes with message Apr 13 15:32:44 kvmtest kernel: libvirtd[8263]: segfault at 4 ip 00000039d7223fc0 sp 00007fa6fbc29a88 error 6 in libdbus-1.so.3.4.0[39d7200000+3c000] in system log, sometimes without any message. (When running the same script from command line it worked for 1000 executions without noticing a problem). When connecting to qemu+tcp:///system using credentials (explained later): <? libvirt_connect($uri,true,Array(VIR_CRED_AUTHNAME=>"fred",VIR_CRED_PASSPHRASE=>"fred")); ?> It works but httpd processes open a lot pipes and after a few hours with page refreshing every 10 sec I even ran into error: [Mon Apr 13 02:40:26 2009] [error] [client 10.38.25.152] PHP Warning: libvirt_connect() unable to make pipe: Too many open files Next issue I am not sure about is callbacks. I need them mainly for authentication. As PHP is intended to run the whole script non-interactively at most times, I've created this solution. When calling the libvirt_connect() PHP function you can provide list of credentials in form of an array Array(VIR_CRED_AUTHNAME=>"fred",VIR_CRED_PASSPHRASE=>"fred"). This PHP array is parsed to C array and this prepopulated array is passed to my authentication callback function. That function receives the requested credential, looks it up in the provided array and returns it. I think this suits more the PHP nature but in future I may provide real callback solution. Little trouble is that libvirt free()s the values returned by callback so I need to copy them to malloc()ed memory. I am not sure about multithread safety of callback functions but I think that if the function only obtains the parameters via cbdata and operates only with them then it should be safe. However sometimes I get error: [Mon Apr 13 14:19:48 2009] [error] [client 10.38.25.152] PHP Warning: libvirt_connect(): Failed to collect auth credentials and I need to restart http daemon for a few times to make it work again. The last trouble is with error callback, where I am not sure whether it is thread safe. I need to call PHP function for reporting error, storing the error in "global" variable for next use and the PHP may even terminate the whole processing of the request... And if you are still reading then I can point you to http://phplibvirt.cybersales.cz/ where you can download the source code and browse the documentation - you can find there list of implemented functions and brief instructions how to install the extension. But be warned it really can crash your libvirt and maybe apache! For completeness: I use Fedora 10 with some rawhide updates: httpd-2.2.11-6.x86_64 php-5.2.9-1.fc11.x86_64 php-devel-5.2.9-1.fc11.x86_64 libvirt-0.6.0-2.fc11.x86_64 qemu-0.9.1-12.fc11.x86_64 libvirt-devel-0.6.0-2.fc11.x86_64 php-cli-5.2.9-1.fc11.x86_64 kvm-83-5.fc10.x86_64 php-common-5.2.9-1.fc11.x86_64 Radek

On Tue, Apr 14, 2009 at 03:56:28PM +0200, Radek Hladik wrote:
When running from command line everything seems to work fine. This is of course singlethreaded and potential resource leaks need not to cause a problem. However when running from webserver, the result is much more interesting. When connecting to qemu:///system (via local socket) using <? libvirt_connect($uri,true); ?> I can crash the libvirt daemon after cca 10 page reloads - sometimes with message Apr 13 15:32:44 kvmtest kernel: libvirtd[8263]: segfault at 4 ip 00000039d7223fc0 sp 00007fa6fbc29a88 error 6 in libdbus-1.so.3.4.0[39d7200000+3c000] in system log, sometimes without any message.
Crashing the libvirtd daemon is not your fault ! It is supposed to be completely robust against whatever bad stuff the client may throw at it. So even if the client has a bug, then it shouldn't crash the daemon. If possible it'd be helpful if you can get a stack trace from the daemon. You can do this by stopping the demon '/etc/init.d/libvirtd stop' and then running it under GDB directly 'gdb /usr/sbin/libvirtd' make sure you have libvirt-debuginfo RPM installed if using Fedora, or have built with '-g' debug option if built manually.
When connecting to qemu+tcp:///system using credentials (explained later): <? libvirt_connect($uri,true,Array(VIR_CRED_AUTHNAME=>"fred",VIR_CRED_PASSPHRASE=>"fred")); ?>
It works but httpd processes open a lot pipes and after a few hours with page refreshing every 10 sec I even ran into error: [Mon Apr 13 02:40:26 2009] [error] [client 10.38.25.152] PHP Warning: libvirt_connect() unable to make pipe: Too many open files
This is interesting. It suggests that the virConnectPtr object you have created is not being relased. Each object will require at least 2 file descriptors, so if they're not released, then either the libvirtd daemon, or the PHP client, or the OS as a whole will run out of file descriptors and start showing this error message.
And if you are still reading then I can point you to http://phplibvirt.cybersales.cz/ where you can download the source code and browse the documentation - you can find there list of implemented functions and brief instructions how to install the extension. But be warned it really can crash your libvirt and maybe apache! For completeness: I use Fedora 10 with some rawhide updates: httpd-2.2.11-6.x86_64 php-5.2.9-1.fc11.x86_64 php-devel-5.2.9-1.fc11.x86_64 libvirt-0.6.0-2.fc11.x86_64 qemu-0.9.1-12.fc11.x86_64 libvirt-devel-0.6.0-2.fc11.x86_64
I'd recommend updating to the latest libvirt 0.6.2 release that should be available in rawhide now. It fixed quite a few crashes & leaks, which you may be unlucky enough to be hitting Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On Tue, Apr 14, 2009 at 04:00:18PM +0100, Daniel P. Berrange wrote:
On Tue, Apr 14, 2009 at 03:56:28PM +0200, Radek Hladik wrote:
It works but httpd processes open a lot pipes and after a few hours with page refreshing every 10 sec I even ran into error: [Mon Apr 13 02:40:26 2009] [error] [client 10.38.25.152] PHP Warning: libvirt_connect() unable to make pipe: Too many open files
This is interesting. It suggests that the virConnectPtr object you have created is not being relased. Each object will require at least 2 file descriptors, so if they're not released, then either the libvirtd daemon, or the PHP client, or the OS as a whole will run out of file descriptors and start showing this error message.
0.6.0 has file descriptors leaks on, so yes this may be solved by an update too. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/

On Tue, Apr 14, 2009 at 04:00:18PM +0100, Daniel P. Berrange wrote:
On Tue, Apr 14, 2009 at 03:56:28PM +0200, Radek Hladik wrote:
When running from command line everything seems to work fine. This is of course singlethreaded and potential resource leaks need not to cause a problem. However when running from webserver, the result is much more interesting. When connecting to qemu:///system (via local socket) using <? libvirt_connect($uri,true); ?> I can crash the libvirt daemon after cca 10 page reloads - sometimes with message Apr 13 15:32:44 kvmtest kernel: libvirtd[8263]: segfault at 4 ip 00000039d7223fc0 sp 00007fa6fbc29a88 error 6 in libdbus-1.so.3.4.0[39d7200000+3c000] in system log, sometimes without any message.
Crashing the libvirtd daemon is not your fault ! It is supposed to be completely robust against whatever bad stuff the client may throw at it. So even if the client has a bug, then it shouldn't crash the daemon.
If possible it'd be helpful if you can get a stack trace from the daemon. You can do this by stopping the demon '/etc/init.d/libvirtd stop' and then running it under GDB directly
'gdb /usr/sbin/libvirtd'
make sure you have libvirt-debuginfo RPM installed if using Fedora, or have built with '-g' debug option if built manually.
Actually now that I remember, there were a definitely a couple of libvirtd crashing bugs in the 0.6.0 release. So well worth upgrading before trying to debug this in any detail
httpd-2.2.11-6.x86_64 php-5.2.9-1.fc11.x86_64 php-devel-5.2.9-1.fc11.x86_64 libvirt-0.6.0-2.fc11.x86_64 qemu-0.9.1-12.fc11.x86_64 libvirt-devel-0.6.0-2.fc11.x86_64
I'd recommend updating to the latest libvirt 0.6.2 release that should be available in rawhide now. It fixed quite a few crashes & leaks, which you may be unlucky enough to be hitting
Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Daniel P. Berrange napsal(a):
On Tue, Apr 14, 2009 at 04:00:18PM +0100, Daniel P. Berrange wrote:
On Tue, Apr 14, 2009 at 03:56:28PM +0200, Radek Hladik wrote:
When running from command line everything seems to work fine. This is of course singlethreaded and potential resource leaks need not to cause a problem. However when running from webserver, the result is much more interesting. When connecting to qemu:///system (via local socket) using <? libvirt_connect($uri,true); ?> I can crash the libvirt daemon after cca 10 page reloads - sometimes with message Apr 13 15:32:44 kvmtest kernel: libvirtd[8263]: segfault at 4 ip 00000039d7223fc0 sp 00007fa6fbc29a88 error 6 in libdbus-1.so.3.4.0[39d7200000+3c000] in system log, sometimes without any message. Crashing the libvirtd daemon is not your fault ! It is supposed to be completely robust against whatever bad stuff the client may throw at it. So even if the client has a bug, then it shouldn't crash the daemon.
If possible it'd be helpful if you can get a stack trace from the daemon. You can do this by stopping the demon '/etc/init.d/libvirtd stop' and then running it under GDB directly
'gdb /usr/sbin/libvirtd'
make sure you have libvirt-debuginfo RPM installed if using Fedora, or have built with '-g' debug option if built manually.
Actually now that I remember, there were a definitely a couple of libvirtd crashing bugs in the 0.6.0 release. So well worth upgrading before trying to debug this in any detail
I upgraded to libvirt 0.6.2 from rawhide - I had to compile it from SRPM because RPM version reported some selinux undefined symbol. But my test system is hybrid of F10, F11 and rawhide :-). On the first view things looks a lot better now. When connecting locally I have not been able to crash the daemon for cca 100 reloads and when connecting via qemu+tcp the count of open pipes seems to be constant. I have cca 15 httpd processes running and they have total 42 handles to pipes open and there are only two unique pipe numbers. This number did not change for last cca 30 minutes. I am watching this in /proc so these pipes need not to be related to libvirt at all. I've updated the documentation - added note that version 0.6.2 is required :-) - and modified the code to refuse to connect when version is lower that 0.6.2. I've put the modified version to http://phplibvirt.cybersales.cz/ as v0.2.1 APLHA. I will test it a little more and I'll see if this version would be safe to be tested by others. Radek

On Tue, Apr 14, 2009 at 07:14:32PM +0200, Radek Hladik wrote:
Daniel P. Berrange napsal(a): I upgraded to libvirt 0.6.2 from rawhide - I had to compile it from SRPM because RPM version reported some selinux undefined symbol. But my test system is hybrid of F10, F11 and rawhide :-). On the first view things looks a lot better now. When connecting locally I have not been able to crash the daemon for cca 100 reloads and when connecting via qemu+tcp the count of open pipes seems to be constant. I have cca 15 httpd processes running and they have total 42 handles to pipes open and there are only two unique pipe numbers. This number did not change for last cca 30 minutes. I am watching this in /proc so these pipes need not to be related to libvirt at all.
Good.
I've updated the documentation - added note that version 0.6.2 is required :-) - and modified the code to refuse to connect when version is lower that 0.6.2. I've put the modified version to
I would avoid this. Don't put a hard limitiation like this in the software, the only hard requirement should be feature based, it's perfectly possible to ship 0.6.0 with a set of patches fixing the bugs you experienced. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/

Daniel Veillard napsal(a):
On Tue, Apr 14, 2009 at 07:14:32PM +0200, Radek Hladik wrote:
Daniel P. Berrange napsal(a): I've updated the documentation - added note that version 0.6.2 is required :-) - and modified the code to refuse to connect when version is lower that 0.6.2. I've put the modified version to
I would avoid this. Don't put a hard limitiation like this in the software, the only hard requirement should be feature based, it's perfectly possible to ship 0.6.0 with a set of patches fixing the bugs you experienced.
Daniel
I will change it to warning then and I will put a lot of warnings on the page and into the documentation. I do not like such hard limitations too but if the situation is that I am able to crash the daemon very easily, (almost with default example code), I want to be careful. I do not want to end up with users downloading the code, running a few examples and crashing their daemon just because they didn't read the docs :-) Radek

On Wed, Apr 15, 2009 at 02:10:20PM +0200, Radek Hladik wrote:
Daniel Veillard napsal(a):
On Tue, Apr 14, 2009 at 07:14:32PM +0200, Radek Hladik wrote:
Daniel P. Berrange napsal(a): I've updated the documentation - added note that version 0.6.2 is required :-) - and modified the code to refuse to connect when version is lower that 0.6.2. I've put the modified version to
I would avoid this. Don't put a hard limitiation like this in the software, the only hard requirement should be feature based, it's perfectly possible to ship 0.6.0 with a set of patches fixing the bugs you experienced.
Daniel
I will change it to warning then and I will put a lot of warnings on the page and into the documentation. I do not like such hard limitations too but if the situation is that I am able to crash the daemon very easily, (almost with default example code), I want to be careful. I do not want to end up with users downloading the code, running a few examples and crashing their daemon just because they didn't read the docs :-)
0.6.0 as shipped should not be installed anywhere, there should have been updates where it was pushed, though it's hard to coordinate. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/

On Tue, Apr 14, 2009 at 03:56:28PM +0200, Radek Hladik wrote:
So one thing is that every memory allocated by libvirt (using malloc()) I need to copy to memory allocated using emalloc(). Also every memory, that would be free()ed by libvirt I need to malloc() and copy from emmaloc()ed memory. The first one is done by simple macro:
#define RECREATE_STRING_WITH_E(str_out, str_in) \ str_out = estrndup(str_in, strlen(str_in)); \ free(str_in);
Yeah, that's painful, but there is no good workaround. Trying to export an api to register new memory allocation routines for the library is usually unpractical because it becomes a global library setting. At least in the libxml2 case where I added that's why it's actually hard to use in practice except very controlled environments.
The second case I encountered only on authentication callback and I use simple strndup() to copy it for libvirt.
When running from command line everything seems to work fine. This is of course singlethreaded and potential resource leaks need not to cause a problem. However when running from webserver, the result is much more interesting. When connecting to qemu:///system (via local socket) using <? libvirt_connect($uri,true); ?> I can crash the libvirt daemon after cca 10 page reloads - sometimes with message Apr 13 15:32:44 kvmtest kernel: libvirtd[8263]: segfault at 4 ip 00000039d7223fc0 sp 00007fa6fbc29a88 error 6 in libdbus-1.so.3.4.0[39d7200000+3c000] in system log, sometimes without any message. (When running the same script from command line it worked for 1000 executions without noticing a problem).
When connecting to qemu+tcp:///system using credentials (explained later): <? libvirt_connect($uri,true,Array(VIR_CRED_AUTHNAME=>"fred",VIR_CRED_PASSPHRASE=>"fred")); ?>
It works but httpd processes open a lot pipes and after a few hours with page refreshing every 10 sec I even ran into error: [Mon Apr 13 02:40:26 2009] [error] [client 10.38.25.152] PHP Warning: libvirt_connect() unable to make pipe: Too many open files
[...]
libvirt-0.6.0-2.fc11.x86_64
Don't look further, you need 0.6.2 for this kind of things to not crash and burn on any serious reggression tests. forget about 0.6.0 !
Next issue I am not sure about is callbacks. I need them mainly for authentication. As PHP is intended to run the whole script non-interactively at most times, I've created this solution. When calling the libvirt_connect() PHP function you can provide list of credentials in form of an array Array(VIR_CRED_AUTHNAME=>"fred",VIR_CRED_PASSPHRASE=>"fred"). This PHP array is parsed to C array and this prepopulated array is passed to my authentication callback function. That function receives the requested credential, looks it up in the provided array and returns it. I think this suits more the PHP nature but in future I may provide real callback solution. Little trouble is that libvirt free()s the values returned by callback so I need to copy them to malloc()ed memory. I am not sure about multithread safety of callback functions but I think that if the function only obtains the parameters via cbdata and operates only with them then it should be safe. However sometimes I get error: [Mon Apr 13 14:19:48 2009] [error] [client 10.38.25.152] PHP Warning: libvirt_connect(): Failed to collect auth credentials and I need to restart http daemon for a few times to make it work again.
No idea here, danpb might know more about this. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
participants (3)
-
Daniel P. Berrange
-
Daniel Veillard
-
Radek Hladik