[libvirt] libvirt-client leaks memory, Ubuntu and Debian-specific
Hello. We faced a very strange leak while using libvirt library in long-running server application for cluster orchestration. Leak does not directly related to libvirt code and exposed only on specific build options (?) and/or system environment (?). Here are the key points: 1. Libvirt client leaks memory while making (RPC) calls to a server. I mean that RSS memory usage shown by ps, top, etc. indefinitely grows (plz check an attachment ps.log). Test app attached. 2. Leak detected on Debian and Ubuntu and absent on Mac OS and Gentoo, so it's exactly an environment or build problem. 3. Valgrind does not see the leak. From valgrind's point of view from start to finish application constantly consumes 110kb of memory (while ps shows multiple megabytes) and does not contain any leaks. 4. Logging activity of virMalloc, virRealloc, etc. functions does not show anything: as expected, all allocated memory is correctly freed, so it's definitely not a bug of code (we tested that before recognized that problem is distrib/platform specific). Some useful logs and test code attached. I think digging build options and legoing system libraries will help us to beat the problem but it would be nice if someone already had a working solution. Thank you for help.
Hello. We did some progress in our search, here is there news: 1. Leak is NOT platform dependent, we tested different versions on different platforms and made a wrong conclusion. 2. Leak originated somewhere in changes between versions 1.0.1 and v 1.0.2. 15.02.2013, в 22:20, Igor Lukyanov <igor@lukyanov.org> написал(а):
Hello. We faced a very strange leak while using libvirt library in long-running server application for cluster orchestration. Leak does not directly related to libvirt code and exposed only on specific build options (?) and/or system environment (?).
Here are the key points: 1. Libvirt client leaks memory while making (RPC) calls to a server. I mean that RSS memory usage shown by ps, top, etc. indefinitely grows (plz check an attachment ps.log). Test app attached.
2. Leak detected on Debian and Ubuntu and absent on Mac OS and Gentoo, so it's exactly an environment or build problem.
3. Valgrind does not see the leak. From valgrind's point of view from start to finish application constantly consumes 110kb of memory (while ps shows multiple megabytes) and does not contain any leaks.
4. Logging activity of virMalloc, virRealloc, etc. functions does not show anything: as expected, all allocated memory is correctly freed, so it's definitely not a bug of code (we tested that before recognized that problem is distrib/platform specific).
Some useful logs and test code attached. I think digging build options and legoing system libraries will help us to beat the problem but it would be nice if someone already had a working solution. Thank you for help.
<leak.c><ps.log><valgrind.log>
Commit 18937c3ae0990b4417a43aa07a2c35aaf8cb6ec2 introduced the memory leak when client->msg.fds is copied to thecall->msg and then never freed. --- src/rpc/virnetclient.c | 1 + 1 file changed, 1 insertion(+) diff --git a/src/rpc/virnetclient.c b/src/rpc/virnetclient.c index 4efa578..bfa1624 100644 --- a/src/rpc/virnetclient.c +++ b/src/rpc/virnetclient.c @@ -1182,6 +1182,7 @@ virNetClientIOWriteMessage(virNetClientPtr client, } thecall->msg->donefds = 0; thecall->msg->bufferOffset = thecall->msg->bufferLength = 0; + VIR_FREE(thecall->msg->fds); VIR_FREE(thecall->msg->buffer); if (thecall->expectReply) thecall->mode = VIR_NET_CLIENT_MODE_WAIT_RX; -- 1.7.10.4
Commit 18937c3ae0990b4417a43aa07a2c35aaf8cb6ec2 introduced the memory leak when client->msg.fds is moved to thecall->msg and then never freed. --- Hello, Please ignore previous patch with incorrent formatting. src/rpc/virnetclient.c | 1 + 1 file changed, 1 insertion(+) diff --git a/src/rpc/virnetclient.c b/src/rpc/virnetclient.c index 4efa578..bfa1624 100644 --- a/src/rpc/virnetclient.c +++ b/src/rpc/virnetclient.c @@ -1182,6 +1182,7 @@ virNetClientIOWriteMessage(virNetClientPtr client, } thecall->msg->donefds = 0; thecall->msg->bufferOffset = thecall->msg->bufferLength = 0; + VIR_FREE(thecall->msg->fds); VIR_FREE(thecall->msg->buffer); if (thecall->expectReply) thecall->mode = VIR_NET_CLIENT_MODE_WAIT_RX; -- 1.7.10.4
On 18.02.2013 12:01, Sergey Fionov wrote:
Commit 18937c3ae0990b4417a43aa07a2c35aaf8cb6ec2 introduced the memory leak when client->msg.fds is moved to thecall->msg and then never freed. --- Hello, Please ignore previous patch with incorrent formatting.
src/rpc/virnetclient.c | 1 + 1 file changed, 1 insertion(+)
diff --git a/src/rpc/virnetclient.c b/src/rpc/virnetclient.c index 4efa578..bfa1624 100644 --- a/src/rpc/virnetclient.c +++ b/src/rpc/virnetclient.c @@ -1182,6 +1182,7 @@ virNetClientIOWriteMessage(virNetClientPtr client, } thecall->msg->donefds = 0; thecall->msg->bufferOffset = thecall->msg->bufferLength = 0; + VIR_FREE(thecall->msg->fds); VIR_FREE(thecall->msg->buffer); if (thecall->expectReply) thecall->mode = VIR_NET_CLIENT_MODE_WAIT_RX;
ACKed and pushed. Fortunately, the commit that introduced this is not present in any of our stable branches, so we don't need to backport it. Michal
On 15.02.2013 19:20, Igor Lukyanov wrote:
Hello. We faced a very strange leak while using libvirt library in long-running server application for cluster orchestration. Leak does not directly related to libvirt code and exposed only on specific build options (?) and/or system environment (?).
Here are the key points: 1. Libvirt client leaks memory while making (RPC) calls to a server. I mean that RSS memory usage shown by ps, top, etc. indefinitely grows (plz check an attachment ps.log). Test app attached.
2. Leak detected on Debian and Ubuntu and absent on Mac OS and Gentoo, so it's exactly an environment or build problem.
3. Valgrind does not see the leak. From valgrind's point of view from start to finish application constantly consumes 110kb of memory (while ps shows multiple megabytes) and does not contain any leaks.
4. Logging activity of virMalloc, virRealloc, etc. functions does not show anything: as expected, all allocated memory is correctly freed, so it's definitely not a bug of code (we tested that before recognized that problem is distrib/platform specific).
Some useful logs and test code attached. I think digging build options and legoing system libraries will help us to beat the problem but it would be nice if someone already had a working solution. Thank you for help.
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
I don't think there's any leak. I mean, not the true one. From your attached valgrind output, and from my own try, I see the only libvirt leak is the one by virInitialize() where init of underlying libraries (like gnutls) takes place. Unfortunately, we cannot de-init them (e.g. by introducing virDeinit API) since the mgmt application can still want to use the libraries. For more info see discussion in: http://www.redhat.com/archives/libvir-list/2011-August/msg01127.html Michal
participants (3)
-
Igor Lukyanov -
Michal Privoznik -
Sergey Fionov