[libvirt] F17's libvirt takes comments into LIBVIRTD_ARGS
On Tue, May 15, 2012 at 04:16:11PM +0800, Shu Ming wrote:
On 2012-5-14 7:30, Dan Kenigsberg wrote:
On Sun, May 13, 2012 at 11:51:48PM +0800, Shu Ming wrote:
Hi, Recently, I found that my host in engine was always in a "unassigned state" after the host node was installed. After looking into the vdsm.log, it seemed that vdsm failed to call libvirt as an error, "libvirtError: Cannot write data: Broken pipe". When I started virsh in the host node at that time, a warning was given "WARNING: no socket to connect to" and core dumped with "virsh net-list". It looks like that no right socket was created for virsh to connect to libvirtd. Any comments about this problem? The followings are my steps in the node:
[root@ovirt-node1 ~]# rpm -qa |grep vdsm vdsm-cli-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-python-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-hook-vhostmd-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-4.9.6-0.183.git107644d.fc16.shuming1336622293.x86_64 vdsm-reg-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-debug-plugin-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-hook-faqemu-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-bootstrap-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch [root@ovirt-node1 ~]# [root@ovirt-node1 ~]# ps -ef |grep libvirt
libvirt-daemon-0.9.11-1.fc17.x86_64 libvirt-daemon-config-nwfilter-0.9.11-1.fc17.x86_64 libvirt-client-0.9.11-1.fc17.x86_64 libvirt-daemon-config-network-0.9.11-1.fc17.x86_64 libvirt-python-0.9.11-1.fc17.x86_64
[root@ovirt-node1 ~]# virsh net-list WARNING: no socket to connect to Segmentation fault I think that merits a libvirt bug. please attach strace output to bugzilla.
[root@ovirt-node1 ~]#
[root@ovirt-node1 ~]# ps -ef |grep vdsm root 1299 1 0 23:10 ? 00:00:00 /usr/sbin/libvirtd --listen # by vdsm The command line of libvirt process is very odd - the comment that vdsm puts into /etc/sysconfig/libvirtd is somehow taken verbatim. That's bad, and may be related to Fedora 17's systemd services. Try to remove the comment and restart libvirtd to see if this is the case. The comment come from
[root@ovirt-node1 ~]# cat /etc/sysconfig/libvirtd:
I know that (see my text above). However, in F16 and before, comments have been stripped before being passed to commandline. Have you tested if all is well when the commment is removed? Let's see what our friends in libvir-list think. Dan.
On Wed, May 16, 2012 at 01:46:43PM +0300, Dan Kenigsberg wrote:
On Tue, May 15, 2012 at 04:16:11PM +0800, Shu Ming wrote:
On 2012-5-14 7:30, Dan Kenigsberg wrote:
On Sun, May 13, 2012 at 11:51:48PM +0800, Shu Ming wrote:
Hi, Recently, I found that my host in engine was always in a "unassigned state" after the host node was installed. After looking into the vdsm.log, it seemed that vdsm failed to call libvirt as an error, "libvirtError: Cannot write data: Broken pipe". When I started virsh in the host node at that time, a warning was given "WARNING: no socket to connect to" and core dumped with "virsh net-list". It looks like that no right socket was created for virsh to connect to libvirtd. Any comments about this problem? The followings are my steps in the node:
[root@ovirt-node1 ~]# rpm -qa |grep vdsm vdsm-cli-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-python-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-hook-vhostmd-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-4.9.6-0.183.git107644d.fc16.shuming1336622293.x86_64 vdsm-reg-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-debug-plugin-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-hook-faqemu-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-bootstrap-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch [root@ovirt-node1 ~]# [root@ovirt-node1 ~]# ps -ef |grep libvirt
libvirt-daemon-0.9.11-1.fc17.x86_64 libvirt-daemon-config-nwfilter-0.9.11-1.fc17.x86_64 libvirt-client-0.9.11-1.fc17.x86_64 libvirt-daemon-config-network-0.9.11-1.fc17.x86_64 libvirt-python-0.9.11-1.fc17.x86_64
[root@ovirt-node1 ~]# virsh net-list WARNING: no socket to connect to Segmentation fault I think that merits a libvirt bug. please attach strace output to bugzilla.
[root@ovirt-node1 ~]#
[root@ovirt-node1 ~]# ps -ef |grep vdsm root 1299 1 0 23:10 ? 00:00:00 /usr/sbin/libvirtd --listen # by vdsm The command line of libvirt process is very odd - the comment that vdsm puts into /etc/sysconfig/libvirtd is somehow taken verbatim. That's bad, and may be related to Fedora 17's systemd services. Try to remove the comment and restart libvirtd to see if this is the case. The comment come from
[root@ovirt-node1 ~]# cat /etc/sysconfig/libvirtd:
I know that (see my text above). However, in F16 and before, comments have been stripped before being passed to commandline. Have you tested if all is well when the commment is removed?
Let's see what our friends in libvir-list think.
In F16 we were using the shell to launch libvirtd, in F17 we are using systemd. systemd has stricter requirements around the syntax of /etc/sysconfig/libvirtd file. Specifically it treats the entire line as the value. FYI I add a BZ for this against vdsm along with a suggested solution https://bugzilla.redhat.com/show_bug.cgi?id=821867 Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, May 16, 2012 at 11:56:36AM +0100, Daniel P. Berrange wrote:
On Wed, May 16, 2012 at 01:46:43PM +0300, Dan Kenigsberg wrote:
On Tue, May 15, 2012 at 04:16:11PM +0800, Shu Ming wrote: I know that (see my text above). However, in F16 and before, comments have been stripped before being passed to commandline. Have you tested if all is well when the commment is removed?
Let's see what our friends in libvir-list think.
In F16 we were using the shell to launch libvirtd, in F17 we are using systemd. systemd has stricter requirements around the syntax of /etc/sysconfig/libvirtd file. Specifically it treats the entire line as the value.
FYI I add a BZ for this against vdsm along with a suggested solution
Oh and there is also a libvirt problem here, because we should not have allowed these bogus argv to be passed - we should have raised an error and exited Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 2012-5-16 18:46, Dan Kenigsberg wrote:
On 2012-5-14 7:30, Dan Kenigsberg wrote:
On Sun, May 13, 2012 at 11:51:48PM +0800, Shu Ming wrote:
Hi, Recently, I found that my host in engine was always in a "unassigned state" after the host node was installed. After looking into the vdsm.log, it seemed that vdsm failed to call libvirt as an error, "libvirtError: Cannot write data: Broken pipe". When I started virsh in the host node at that time, a warning was given "WARNING: no socket to connect to" and core dumped with "virsh net-list". It looks like that no right socket was created for virsh to connect to libvirtd. Any comments about this problem? The followings are my steps in the node:
[root@ovirt-node1 ~]# rpm -qa |grep vdsm vdsm-cli-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-python-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-hook-vhostmd-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-4.9.6-0.183.git107644d.fc16.shuming1336622293.x86_64 vdsm-reg-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-debug-plugin-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-hook-faqemu-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-bootstrap-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch [root@ovirt-node1 ~]# [root@ovirt-node1 ~]# ps -ef |grep libvirt
libvirt-daemon-0.9.11-1.fc17.x86_64 libvirt-daemon-config-nwfilter-0.9.11-1.fc17.x86_64 libvirt-client-0.9.11-1.fc17.x86_64 libvirt-daemon-config-network-0.9.11-1.fc17.x86_64 libvirt-python-0.9.11-1.fc17.x86_64
[root@ovirt-node1 ~]# virsh net-list WARNING: no socket to connect to Segmentation fault I think that merits a libvirt bug. please attach strace output to bugzilla.
[root@ovirt-node1 ~]#
[root@ovirt-node1 ~]# ps -ef |grep vdsm root 1299 1 0 23:10 ? 00:00:00 /usr/sbin/libvirtd --listen # by vdsm The command line of libvirt process is very odd - the comment that vdsm puts into /etc/sysconfig/libvirtd is somehow taken verbatim. That's bad, and may be related to Fedora 17's systemd services. Try to remove the comment and restart libvirtd to see if this is the case. The comment come from
[root@ovirt-node1 ~]# cat /etc/sysconfig/libvirtd: I know that (see my text above). However, in F16 and before, comments have been stripped before being passed to commandline. Have you tested if all is well when
On Tue, May 15, 2012 at 04:16:11PM +0800, Shu Ming wrote: the commment is removed?
I removed the "# by vdsm " line from the config file. And restarted the libvirtd and vdsmd service. But no luck to make "virsh net-list" successful, still got "Segmentation fault", while "virsh -c qemu:///system -r" worked. virsh # net-list Segmentation fault [root@ovirt-node1 ~]# virsh net-list Segmentation fault [root@ovirt-node1 ~]# virsh -c qemu:///system Please enter your authentication name: ^C [root@ovirt-node1 ~]# virsh -c qemu:///system -r Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh > net-list Name State Autostart ----------------------------------------- vdsm-ovirtmgmt active yes virsh >
Let's see what our friends in libvir-list think.
Dan.
-- Shu Ming<shuming@linux.vnet.ibm.com> IBM China Systems and Technology Laboratory
On Wed, May 16, 2012 at 11:05:16PM +0800, Shu Ming wrote:
On 2012-5-16 18:46, Dan Kenigsberg wrote:
On 2012-5-14 7:30, Dan Kenigsberg wrote:
On Sun, May 13, 2012 at 11:51:48PM +0800, Shu Ming wrote:
Hi, Recently, I found that my host in engine was always in a "unassigned state" after the host node was installed. After looking into the vdsm.log, it seemed that vdsm failed to call libvirt as an error, "libvirtError: Cannot write data: Broken pipe". When I started virsh in the host node at that time, a warning was given "WARNING: no socket to connect to" and core dumped with "virsh net-list". It looks like that no right socket was created for virsh to connect to libvirtd. Any comments about this problem? The followings are my steps in the node:
[root@ovirt-node1 ~]# rpm -qa |grep vdsm vdsm-cli-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-python-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-hook-vhostmd-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-4.9.6-0.183.git107644d.fc16.shuming1336622293.x86_64 vdsm-reg-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-debug-plugin-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-hook-faqemu-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-bootstrap-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch [root@ovirt-node1 ~]# [root@ovirt-node1 ~]# ps -ef |grep libvirt
libvirt-daemon-0.9.11-1.fc17.x86_64 libvirt-daemon-config-nwfilter-0.9.11-1.fc17.x86_64 libvirt-client-0.9.11-1.fc17.x86_64 libvirt-daemon-config-network-0.9.11-1.fc17.x86_64 libvirt-python-0.9.11-1.fc17.x86_64
[root@ovirt-node1 ~]# virsh net-list WARNING: no socket to connect to Segmentation fault I think that merits a libvirt bug. please attach strace output to bugzilla.
[root@ovirt-node1 ~]#
[root@ovirt-node1 ~]# ps -ef |grep vdsm root 1299 1 0 23:10 ? 00:00:00 /usr/sbin/libvirtd --listen # by vdsm The command line of libvirt process is very odd - the comment that vdsm puts into /etc/sysconfig/libvirtd is somehow taken verbatim. That's bad, and may be related to Fedora 17's systemd services. Try to remove the comment and restart libvirtd to see if this is the case. The comment come from
[root@ovirt-node1 ~]# cat /etc/sysconfig/libvirtd: I know that (see my text above). However, in F16 and before, comments have been stripped before being passed to commandline. Have you tested if all is well when
On Tue, May 15, 2012 at 04:16:11PM +0800, Shu Ming wrote: the commment is removed?
I removed the "# by vdsm " line from the config file. And restarted the libvirtd and vdsmd service. But no luck to make "virsh net-list" successful, still got "Segmentation fault", while "virsh -c qemu:///system -r" worked.
Thanks. Please strace that segfault, there's a libvirt bug lying there. And thanks again for finding the vdsm/libvirt configuration problem in F17.
On Wed, May 16, 2012 at 06:08:51PM +0300, Dan Kenigsberg wrote:
On Wed, May 16, 2012 at 11:05:16PM +0800, Shu Ming wrote:
On 2012-5-16 18:46, Dan Kenigsberg wrote:
On 2012-5-14 7:30, Dan Kenigsberg wrote:
On Sun, May 13, 2012 at 11:51:48PM +0800, Shu Ming wrote:
Hi, Recently, I found that my host in engine was always in a "unassigned state" after the host node was installed. After looking into the vdsm.log, it seemed that vdsm failed to call libvirt as an error, "libvirtError: Cannot write data: Broken pipe". When I started virsh in the host node at that time, a warning was given "WARNING: no socket to connect to" and core dumped with "virsh net-list". It looks like that no right socket was created for virsh to connect to libvirtd. Any comments about this problem? The followings are my steps in the node:
[root@ovirt-node1 ~]# rpm -qa |grep vdsm vdsm-cli-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-python-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-hook-vhostmd-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-4.9.6-0.183.git107644d.fc16.shuming1336622293.x86_64 vdsm-reg-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-debug-plugin-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-hook-faqemu-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch vdsm-bootstrap-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch [root@ovirt-node1 ~]# [root@ovirt-node1 ~]# ps -ef |grep libvirt
libvirt-daemon-0.9.11-1.fc17.x86_64 libvirt-daemon-config-nwfilter-0.9.11-1.fc17.x86_64 libvirt-client-0.9.11-1.fc17.x86_64 libvirt-daemon-config-network-0.9.11-1.fc17.x86_64 libvirt-python-0.9.11-1.fc17.x86_64
[root@ovirt-node1 ~]# virsh net-list WARNING: no socket to connect to Segmentation fault I think that merits a libvirt bug. please attach strace output to bugzilla.
[root@ovirt-node1 ~]#
[root@ovirt-node1 ~]# ps -ef |grep vdsm root 1299 1 0 23:10 ? 00:00:00 /usr/sbin/libvirtd --listen # by vdsm The command line of libvirt process is very odd - the comment that vdsm puts into /etc/sysconfig/libvirtd is somehow taken verbatim. That's bad, and may be related to Fedora 17's systemd services. Try to remove the comment and restart libvirtd to see if this is the case. The comment come from
[root@ovirt-node1 ~]# cat /etc/sysconfig/libvirtd: I know that (see my text above). However, in F16 and before, comments have been stripped before being passed to commandline. Have you tested if all is well when
On Tue, May 15, 2012 at 04:16:11PM +0800, Shu Ming wrote: the commment is removed?
I removed the "# by vdsm " line from the config file. And restarted the libvirtd and vdsmd service. But no luck to make "virsh net-list" successful, still got "Segmentation fault", while "virsh -c qemu:///system -r" worked.
Thanks. Please strace that segfault, there's a libvirt bug lying there. And thanks again for finding the vdsm/libvirt configuration problem in F17.
Specifically do the following # debuginfo-install libvirt # gdb --args virsh net-list (gdb) run ...wait for crash.. (gdb) thread apply all backtrace Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 2012-5-16 23:35, Daniel P. Berrange wrote:
On Wed, May 16, 2012 at 06:08:51PM +0300, Dan Kenigsberg wrote:
On Wed, May 16, 2012 at 11:05:16PM +0800, Shu Ming wrote:
On 2012-5-16 18:46, Dan Kenigsberg wrote:
On 2012-5-14 7:30, Dan Kenigsberg wrote:
On Sun, May 13, 2012 at 11:51:48PM +0800, Shu Ming wrote: > Hi, > Recently, I found that my host in engine was always in a > "unassigned state" after the host node was installed. After looking > into the vdsm.log, it seemed that vdsm failed to call libvirt as an > error, "libvirtError: Cannot write data: Broken pipe". When I > started virsh in the host node at that time, a warning was given > "WARNING: no socket to connect to" and core dumped with "virsh > net-list". It looks like that no right socket was created for > virsh to connect to libvirtd. Any comments about this problem? The > followings are my steps in the node: > > [root@ovirt-node1 ~]# rpm -qa |grep vdsm > vdsm-cli-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch > vdsm-python-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch > vdsm-hook-vhostmd-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch > vdsm-4.9.6-0.183.git107644d.fc16.shuming1336622293.x86_64 > vdsm-reg-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch > vdsm-debug-plugin-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch > vdsm-hook-faqemu-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch > vdsm-bootstrap-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch > [root@ovirt-node1 ~]# > [root@ovirt-node1 ~]# ps -ef |grep libvirt > > libvirt-daemon-0.9.11-1.fc17.x86_64 > libvirt-daemon-config-nwfilter-0.9.11-1.fc17.x86_64 > libvirt-client-0.9.11-1.fc17.x86_64 > libvirt-daemon-config-network-0.9.11-1.fc17.x86_64 > libvirt-python-0.9.11-1.fc17.x86_64 > > [root@ovirt-node1 ~]# virsh net-list > WARNING: no socket to connect to > Segmentation fault I think that merits a libvirt bug. please attach strace output to bugzilla.
> [root@ovirt-node1 ~]# > > > [root@ovirt-node1 ~]# ps -ef |grep vdsm > root 1299 1 0 23:10 ? 00:00:00 /usr/sbin/libvirtd > --listen # by vdsm The command line of libvirt process is very odd - the comment that vdsm puts into /etc/sysconfig/libvirtd is somehow taken verbatim. That's bad, and may be related to Fedora 17's systemd services. Try to remove the comment and restart libvirtd to see if this is the case. The comment come from
[root@ovirt-node1 ~]# cat /etc/sysconfig/libvirtd: I know that (see my text above). However, in F16 and before, comments have been stripped before being passed to commandline. Have you tested if all is well when
On Tue, May 15, 2012 at 04:16:11PM +0800, Shu Ming wrote: the commment is removed? I removed the "# by vdsm " line from the config file. And restarted the libvirtd and vdsmd service. But no luck to make "virsh net-list" successful, still got "Segmentation fault", while "virsh -c qemu:///system -r" worked. Thanks. Please strace that segfault, there's a libvirt bug lying there. And thanks again for finding the vdsm/libvirt configuration problem in F17. Specifically do the following
# debuginfo-install libvirt # gdb --args virsh net-list (gdb) run ...wait for crash.. (gdb) thread apply all backtrace
Regards, Daniel [root@ovirt-node1 ~]# debuginfo-install libvirt Loaded plugins: langpacks, presto, refresh-packagekit enabling fedora-debuginfo enabling updates-debuginfo No debuginfo packages available to install [root@ovirt-node1 ~]# debuginfo-install libvirt --releasever=17 Loaded plugins: langpacks, presto, refresh-packagekit enabling fedora-debuginfo enabling updates-debuginfo No debuginfo packages available to install [root@ovirt-node1 ~]#
-- Shu Ming<shuming@linux.vnet.ibm.com> IBM China Systems and Technology Laboratory
On Wed, May 16, 2012 at 11:45:13PM +0800, Shu Ming wrote:
On 2012-5-16 23:35, Daniel P. Berrange wrote:
>>[root@ovirt-node1 ~]# rpm -qa |grep vdsm >>vdsm-cli-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch >>vdsm-python-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch >>vdsm-hook-vhostmd-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch >>vdsm-4.9.6-0.183.git107644d.fc16.shuming1336622293.x86_64 >>vdsm-reg-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch >>vdsm-debug-plugin-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch >>vdsm-hook-faqemu-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch >>vdsm-bootstrap-4.9.6-0.183.git107644d.fc16.shuming1336622293.noarch
[snip]
[root@ovirt-node1 ~]# debuginfo-install libvirt Loaded plugins: langpacks, presto, refresh-packagekit enabling fedora-debuginfo enabling updates-debuginfo No debuginfo packages available to install [root@ovirt-node1 ~]# debuginfo-install libvirt --releasever=17 Loaded plugins: langpacks, presto, refresh-packagekit enabling fedora-debuginfo enabling updates-debuginfo No debuginfo packages available to install [root@ovirt-node1 ~]#
What is your actual distro ? You see to have a wierd mix of Fedora 16 and Fedora 17 packages here, so I presume you've been installing custom built packages, or mixing different Fedora repositories There is definitely a libvirt-debuginfo RPM present in the normal Fedora 17 repositories. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (3)
-
Dan Kenigsberg -
Daniel P. Berrange -
Shu Ming