8:53 p.m.
Hi,
We are encountering a problem of race conditions for getting VNC port
when VM start up.
In a very powerful hypervisor, if we try to start up more than 1 VM
concurrently, then there might be some VMs failed to start up due to VNC
port confilict.
We searched libvirt bugs and found that some one report the problem and
we have give a fix, but the fix can not resolve the problem.
http://osdir.com/ml/libvir-list/2010-05/msg00530.html
https://build.opensuse.org/package/view_file?file=vnc-race-3.patch&packa
ge=libvirt&project=Virtualization:openSUSE11.3
The bitmap test-and-set operation is not atomic, so that there still is
a time-window in which a VNC port could be reserved for two caller.
Your help is highly appreciated!
Thanks,
Guangya Liu
Cloud Developer
Platform Computing
direct: +86-29-87607400-333
www.platform.com <http://www.platform.com/>
9:47 p.m.
On Sun, Nov 07, 2010 at 08:53:32PM +0800, Guangya Liu wrote:
Hi,
We are encountering a problem of race conditions for getting VNC port
when VM start up.
In a very powerful hypervisor, if we try to start up more than 1 VM
concurrently, then there might be some VMs failed to start up due to VNC
port confilict.
We searched libvirt bugs and found that some one report the problem and
we have give a fix, but the fix can not resolve the problem.
http://osdir.com/ml/libvir-list/2010-05/msg00530.html
https://build.opensuse.org/package/view_file?file=vnc-race-3.patch&packa
ge=libvirt&project=Virtualization:openSUSE11.3
This patch is already merged in current GIT repos.
The bitmap test-and-set operation is not atomic, so that there still
is
a time-window in which a VNC port could be reserved for two caller.
The virBitmap APIs for getting & setting don't need to be atomic, because
they are only called when the QEMU driver mutex is held, which ensures
serialization of VM startup.
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
1:21 a.m.
Thanks Daniel for the info.
But I still have a question, since we already make the call
qemudDomainCreate synced, why do we need to introduce a bitmap to
resolve the problem of getting VNC port conflict error? If
qemudDomainCreate is synced, then there should not any problem with VNC
port.
Looking forward to your reply.
static virDomainPtr qemudDomainCreate(virConnectPtr conn, const char
*xml,
unsigned int flags) {
struct qemud_driver *driver = conn->privateData;
virDomainDefPtr def;
virDomainObjPtr vm = NULL;
virDomainPtr dom = NULL;
virDomainEventPtr event = NULL;
virCheckFlags(VIR_DOMAIN_START_PAUSED, NULL);
qemuDriverLock(driver); <==
Thanks,
Guangya
-----Original Message-----
From: Daniel P. Berrange [mailto:berrange@redhat.com]
Sent: Tuesday, November 09, 2010 9:47 PM
To: Guangya Liu
Cc: libvir-list(a)redhat.com
Subject: Re: [libvirt] race condtion in getting VNC port for libvirt
On Sun, Nov 07, 2010 at 08:53:32PM +0800, Guangya Liu wrote:
https://build.opensuse.org/package/view_file?file=vnc-race-3.patch&packa
Hi,
We are encountering a problem of race conditions for getting VNC port
when VM start up.
In a very powerful hypervisor, if we try to start up more than 1 VM
concurrently, then there might be some VMs failed to start up due to
VNC
port confilict.
We searched libvirt bugs and found that some one report the problem
and
we have give a fix, but the fix can not resolve the problem.
ge=libvirt&project=Virtualization:openSUSE11.3
This patch is already merged in current GIT repos.
The bitmap test-and-set operation is not atomic, so that there still
is
a time-window in which a VNC port could be reserved for two caller.
The virBitmap APIs for getting & setting don't need to be atomic,
because
they are only called when the QEMU driver mutex is held, which ensures
serialization of VM startup.
Regards,
Daniel
--
|: Red Hat, Engineering, London -o-
http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o-
http://deltacloud.org :|
|: http://autobuild.org -o-
http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
9505 :|
5:17 a.m.
On Wed, Nov 10, 2010 at 01:21:24AM +0800, Guangya Liu wrote:
Thanks Daniel for the info.
But I still have a question, since we already make the call
qemudDomainCreate synced, why do we need to introduce a bitmap to
resolve the problem of getting VNC port conflict error? If
qemudDomainCreate is synced, then there should not any problem with VNC
port.
Previously, libvirt would call bind() on port numbers until it found
one free, then close() the socket & launch QEMU telling it to open
that port again. Thus there is a race between libvirt finding a free
port with bind()+close() for the first VM, and QEMU actually opening
it, where libvirt could then start looking for another port for a
second VM. The libvirt driver lock doesn't solve that, because the code
involves a separate QEMU process with is outside the context of the
lock
With the bitmap code, libvirt now records that it assigned this
port number to a VM, so when libvirt goes to start a second VM
it won't reuse this port, even if the first QEMU hasn't yet got
around to opening it.
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
5105
days inactive
5107
days old
3 comments
2 participants
participants (2)
-
Daniel P. Berrange -
Guangya Liu