[libvirt] Does libvirt live migration have error handling on port bind failure
Hello, This was the scenario close to 3 years back, libvirt's live migration tries to use ports in ephemeral port range, but has no fallback to use (an)other port(s) when the one it wants is already in use. If some port say 49152 is already used by some application say gluster in our case (gluster as of today also uses 49152-65535), live migration fails because of lack of fallback mechanism in libvirt, that's where gluster had compromised to go with some hack [1] on bug [2] since getting that addressed in libvirt takes more time than it does with gluster. As may releases passed from then in libvirt, I hope now there exist a fallback mechanism for port conflicts in libvirt. Can someone confirm so ? Also It will be greatly appreciable, if someone can tell how the port binding (mostly defense on clash) works with libvirt live migration today ? Sincere Thanks, -- Prasanna
[ oops! apologies, my previous draft miss the links ] Hello, This was the scenario close to 3 years back, libvirt's live migration tries to use ports in ephemeral port range, but has no fallback to use (an)other port(s) when the one it wants is already in use. If some port say 49152 is already used by some application say gluster in our case (gluster as of today also uses 49152-65535), live migration fails because of lack of fallback mechanism in libvirt, that's where gluster had compromised to go with some hack [1] on bug [2] since getting that addressed in libvirt takes more time than it does with gluster. As may releases passed from then in libvirt, I hope now there exist a fallback mechanism for port conflicts in libvirt. Can someone confirm so ? Also It will be greatly appreciable, if someone can tell how the port binding (mostly defense on clash) works with libvirt live migration today ? [1] http://review.gluster.org/#/c/6210/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=1018178 Sincere Thanks, -- Prasanna
Can someone please invest few cycles here ? Thanks, -- Prasanna On Wed, Aug 17, 2016 at 2:32 PM, Prasanna Kalever <pkalever@redhat.com> wrote:
[ oops! apologies, my previous draft miss the links ]
Hello,
This was the scenario close to 3 years back, libvirt's live migration tries to use ports in ephemeral port range, but has no fallback to use (an)other port(s) when the one it wants is already in use.
If some port say 49152 is already used by some application say gluster in our case (gluster as of today also uses 49152-65535), live migration fails because of lack of fallback mechanism in libvirt, that's where gluster had compromised to go with some hack [1] on bug [2] since getting that addressed in libvirt takes more time than it does with gluster.
As may releases passed from then in libvirt, I hope now there exist a fallback mechanism for port conflicts in libvirt.
Can someone confirm so ?
Also It will be greatly appreciable, if someone can tell how the port binding (mostly defense on clash) works with libvirt live migration today ?
[1] http://review.gluster.org/#/c/6210/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=1018178
Sincere Thanks, -- Prasanna
On 08/22/2016 08:08 AM, Prasanna Kalever wrote:
Can someone please invest few cycles here ?
The bulk of the team is at KVM Forum in Toronto with limited time and access, so please try to be patient. John
Thanks, -- Prasanna
On Wed, Aug 17, 2016 at 2:32 PM, Prasanna Kalever <pkalever@redhat.com> wrote:
[ oops! apologies, my previous draft miss the links ]
Hello,
This was the scenario close to 3 years back, libvirt's live migration tries to use ports in ephemeral port range, but has no fallback to use (an)other port(s) when the one it wants is already in use.
If some port say 49152 is already used by some application say gluster in our case (gluster as of today also uses 49152-65535), live migration fails because of lack of fallback mechanism in libvirt, that's where gluster had compromised to go with some hack [1] on bug [2] since getting that addressed in libvirt takes more time than it does with gluster.
As may releases passed from then in libvirt, I hope now there exist a fallback mechanism for port conflicts in libvirt.
Can someone confirm so ?
Also It will be greatly appreciable, if someone can tell how the port binding (mostly defense on clash) works with libvirt live migration today ?
[1] http://review.gluster.org/#/c/6210/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=1018178
Sincere Thanks, -- Prasanna
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 17.08.2016 11:02, Prasanna Kalever wrote:
[ oops! apologies, my previous draft miss the links ]
Hello,
This was the scenario close to 3 years back, libvirt's live migration tries to use ports in ephemeral port range, but has no fallback to use (an)other port(s) when the one it wants is already in use.
If some port say 49152 is already used by some application say gluster in our case (gluster as of today also uses 49152-65535), live migration fails because of lack of fallback mechanism in libvirt, that's where gluster had compromised to go with some hack [1] on bug [2] since getting that addressed in libvirt takes more time than it does with gluster.
I'd say we are pretty quick when it comes to bugfixes but maybe that's just my POV.
As may releases passed from then in libvirt, I hope now there exist a fallback mechanism for port conflicts in libvirt.
Firstly, you can configure whatever port range you want in /etc/libvirt/qemu.conf: #migration_port_min = 49152 #migration_port_max = 49215 Secondly, since 9c9d4d32 (merged Jan 16 2013) libvirt uses this port allocator. It is an internal module that tries to allocate next free port within configured range. So the whole migration then works like this: 1) source client (or daemon in case of p2p migration) contacts remote daemon telling it about intended domain migration 2) destination daemon uses port allocator module to find a free port within configure range 3) destination daemon sends back to the source port number (or an error if none was found) 4) source proceeds with the received port number
Can someone confirm so ?
Also It will be greatly appreciable, if someone can tell how the port binding (mostly defense on clash) works with libvirt live migration today ?
There is none. I mean, there will always be a window between "detecting port free" and "starting qemu which binds to it". One thing that we could do is to bind to the port immediately (thus holding it open without receiving anything on it) and passing it to qemu as it starts. However, who would unbind the port if the source decides not to migrate after step 4) from my example above? Michal
On Tue, Aug 23, 2016 at 4:10 AM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 17.08.2016 11:02, Prasanna Kalever wrote:
[ oops! apologies, my previous draft miss the links ]
Hello,
This was the scenario close to 3 years back, libvirt's live migration tries to use ports in ephemeral port range, but has no fallback to use (an)other port(s) when the one it wants is already in use.
If some port say 49152 is already used by some application say gluster in our case (gluster as of today also uses 49152-65535), live migration fails because of lack of fallback mechanism in libvirt, that's where gluster had compromised to go with some hack [1] on bug [2] since getting that addressed in libvirt takes more time than it does with gluster.
I'd say we are pretty quick when it comes to bugfixes but maybe that's just my POV.
As may releases passed from then in libvirt, I hope now there exist a fallback mechanism for port conflicts in libvirt.
Firstly, you can configure whatever port range you want in /etc/libvirt/qemu.conf:
#migration_port_min = 49152 #migration_port_max = 49215
Secondly, since 9c9d4d32 (merged Jan 16 2013) libvirt uses this port allocator. It is an internal module that tries to allocate next free port within configured range. So the whole migration then works like this:
1) source client (or daemon in case of p2p migration) contacts remote daemon telling it about intended domain migration 2) destination daemon uses port allocator module to find a free port within configure range 3) destination daemon sends back to the source port number (or an error if none was found) 4) source proceeds with the received port number
Can someone confirm so ?
Also It will be greatly appreciable, if someone can tell how the port binding (mostly defense on clash) works with libvirt live migration today ?
There is none. I mean, there will always be a window between "detecting port free" and "starting qemu which binds to it". One thing that we could do is to bind to the port immediately (thus holding it open without receiving anything on it) and passing it to qemu as it starts. However, who would unbind the port if the source decides not to migrate after step 4) from my example above?
Thanks Michal that really helps. So If I understand it right, in the window between destination daemon passing the port (listening) and source (qemu) binding to it, there could be some other external process that uses the same port (race? again I'm not sure how big the window is?) Don't you think libvirt needs a fall back mechanism here, since the port numbers could be in ephemeral port range and are free to be used by any application ? This is just a suggestion, this is how we had implemented our fall back mechanism [1] But now we migrating to a different model something like [2] Please correct me if I took it wrong. [1] http://review.gluster.org/#/c/14235/ [2] http://review.gluster.org/#/c/14613/ Cheers, -- Prasanna
Michal
On 23.08.2016 08:57, Prasanna Kalever wrote:
On Tue, Aug 23, 2016 at 4:10 AM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 17.08.2016 11:02, Prasanna Kalever wrote:
[ oops! apologies, my previous draft miss the links ]
Hello,
Thanks Michal that really helps.
So If I understand it right, in the window between destination daemon passing the port (listening) and source (qemu) binding to it, there could be some other external process that uses the same port (race? again I'm not sure how big the window is?)
Yes. there's this possibility. But then again - these applications you're using should be configured to use different sets of ports. For instance libvirt can use 49152 - 49182, while gluster could use 49183 - 49213. The window is not that long. Maybe my description in the previous e-mail wasn't that clear. At the destination, where the port allocation takes place: 1) libvirt uses its internal module to find a free port 2) libvirt spawns the qemu and tell it to bind to that port 3) libvirt reports back to the migration source host with the port number. So the window is not that big - probably just a couple of milisecs as it's just that part between steps 1 and 2. But there always be some window - even if we implemented what I suggested earlier. That's one of the limitations of kernel API. One simply doesn't detect port free and bind to it atomically.
Don't you think libvirt needs a fall back mechanism here, since the port numbers could be in ephemeral port range and are free to be used by any application ?
You should fix the application then. Or its configuration. We may end up in fallback loop if two applications with the fallback start fighting over the ports. I mean, if it was libvirt who allocates the migration port(socket) for qemu and then just merely pass the FD to it, then libvirt could do a couple of tries to allocate the port (bind socket to it). However, if it is still qemu doing that, we would need to spawn qemu each time we want to try next port. That's not very efficient. Frankly, I have no idea why we don't allocate the port ourselves. I'll ask offline our migration guru and let you know :) Michal
On Wed, Aug 24, 2016 at 2:37 AM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 23.08.2016 08:57, Prasanna Kalever wrote:
On Tue, Aug 23, 2016 at 4:10 AM, Michal Privoznik <mprivozn@redhat.com> wrote:
On 17.08.2016 11:02, Prasanna Kalever wrote:
[ oops! apologies, my previous draft miss the links ]
Hello,
Thanks Michal that really helps.
So If I understand it right, in the window between destination daemon passing the port (listening) and source (qemu) binding to it, there could be some other external process that uses the same port (race? again I'm not sure how big the window is?)
Yes. there's this possibility. But then again - these applications you're using should be configured to use different sets of ports. For instance libvirt can use 49152 - 49182, while gluster could use 49183 - 49213.
From my/our experiences it is better not to take control over ephemeral port ranges. It is very much recommended to leave the choice of port selection for kernel by something like 'bind(0)' And controlling the required port ranges (properties) via local port ranges and local reserved ports (in LINUX)
In linux systems you can find about them in net.ipv4.ip_local_port_range & net.ipv4.ip_local_reserved_ports and in NetBSD {ip.lowportmin, ip.lowportmax} & {ip.anonportmin, ip.anonportmax} If we are bound to it that's wonderful or else at least we should honor the port ranges and reserved ports to avoid clashes with third party applications and better firewall management. Does libvirt honor local port ranges and reserved port range ?
The window is not that long. Maybe my description in the previous e-mail wasn't that clear. At the destination, where the port allocation takes place: 1) libvirt uses its internal module to find a free port 2) libvirt spawns the qemu and tell it to bind to that port 3) libvirt reports back to the migration source host with the port number.
So the window is not that big - probably just a couple of milisecs as it's just that part between steps 1 and 2. But there always be some window - even if we implemented what I suggested earlier. That's one of the limitations of kernel API. One simply doesn't detect port free and bind to it atomically.
Make Sense, still we have window here :-)
Don't you think libvirt needs a fall back mechanism here, since the port numbers could be in ephemeral port range and are free to be used by any application ?
You should fix the application then. Or its configuration. We may end up in fallback loop if two applications with the fallback start fighting over the ports.
I mean, if it was libvirt who allocates the migration port(socket) for qemu and then just merely pass the FD to it, then libvirt could do a couple of tries to allocate the port (bind socket to it). However, if it is still qemu doing that, we would need to spawn qemu each time we want to try next port. That's not very efficient.
We have done something like this now, In our previous implementation glusterd (management demon) used to allocate a free port pass that to brick process (which is glusterfsd) and now glusterfsd use to bind on it, in case in the window some one catch over that port brick used to simply fail. Then we though of implementing fallback mechanism [1], by getting a new port on bind fail for clash/inuse. Then after we came up with a design that respects local port ranges and reserve port ranges [2] which honors everything and even firewall management became very easy for us now. And In this design we have left port binding to glusterfsd (brick process) so there is no window now. I request you to go along the commit messages in the links so that you start appreciating the design.
Frankly, I have no idea why we don't allocate the port ourselves. I'll ask offline our migration guru and let you know :)
Yeah, +1 for this :-) [1] http://review.gluster.org/#/c/14235/ [2] http://review.gluster.org/#/c/14613/ Cheers Michal, -- Prasanna
Michal
participants (3)
-
John Ferlan -
Michal Privoznik -
Prasanna Kalever