[libvirt] Questions about virtlogd
Hello libvirt maintainers, Libvirt implements virtlogd in version 1.3 which now handles logging for QEMU process. I am wondering if it is possible to make it a separate package and maintain stable interfaces for external users? This is related to XSA-180 / CVE-2014-3672 (unrestricted QEMU logging). We are evaluating using virtlogd vs writing our own solution. I believe there are still some open questions on how exactly the integration could be done but let's worry about that later. Thanks Wei. [0] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-3672 [1] http://xenbits.xen.org/xsa/advisory-180.html
On Tue, Jun 07, 2016 at 01:11:53PM +0100, Wei Liu wrote:
Hello libvirt maintainers,
Libvirt implements virtlogd in version 1.3 which now handles logging for QEMU process. I am wondering if it is possible to make it a separate package and maintain stable interfaces for external users?
Ok, so you're essentially asking for us to create a libvirt-logd.so library for talking to virtlogd, which would basically contain the code currently in src/logging/log_manager.c That's certainly possible from a technical POV, but the real question is whether we want to do that from a policy POV, given the greater support implications that has.
This is related to XSA-180 / CVE-2014-3672 (unrestricted QEMU logging). We are evaluating using virtlogd vs writing our own solution. I believe there are still some open questions on how exactly the integration could be done but let's worry about that later.
I must admit I'm not familiar with the division of responsibility for managing QEMU between the Xen provided libxl library(s) and the libvirt libxl driver code. Naively I would expect the libvirt libxl driver code to deal with virtlogd and then configure the Xen libxl library / QEMU accordingly. Your request seems to imply that you will need the Xen libxl library to directly talk to virtlogd instead. Is there any way in which it would be practical for the libvirt libxl driver to talk to virtlogd to acquire the file descriptors to use and pass those file descriptors down to the libxl library ? Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Tue, Jun 07, 2016 at 02:21:17PM +0100, Daniel P. Berrange wrote:
On Tue, Jun 07, 2016 at 01:11:53PM +0100, Wei Liu wrote:
Hello libvirt maintainers,
Libvirt implements virtlogd in version 1.3 which now handles logging for QEMU process. I am wondering if it is possible to make it a separate package and maintain stable interfaces for external users?
Ok, so you're essentially asking for us to create a libvirt-logd.so library for talking to virtlogd, which would basically contain the code currently in src/logging/log_manager.c
Originally I was thinking about have virtlogd - the daemon itself - to be a separate package. That basically means libvirt is not absolutely required for using virtlogd. But from a policy point of view that might not be feasible.
That's certainly possible from a technical POV, but the real question is whether we want to do that from a policy POV, given the greater support implications that has.
There will be support commitment. The interfaces (RPC or library APIs) need to be stable. I understand if this request doesn't align with the support policy. Just knowing the maintainers' opinion on this matter is a good enough starting point for me, which help me evaluate the situation better.
This is related to XSA-180 / CVE-2014-3672 (unrestricted QEMU logging). We are evaluating using virtlogd vs writing our own solution. I believe there are still some open questions on how exactly the integration could be done but let's worry about that later.
I must admit I'm not familiar with the division of responsibility for managing QEMU between the Xen provided libxl library(s) and the libvirt libxl driver code. Naively I would expect the libvirt libxl driver code to deal with virtlogd and then configure the Xen libxl library / QEMU accordingly. Your request seems to imply that you will need the Xen libxl library to directly talk to virtlogd instead.
Is there any way in which it would be practical for the libvirt libxl driver to talk to virtlogd to acquire the file descriptors to use and pass those file descriptors down to the libxl library ?
There are two classes of configurations. For libvirt + libxl, There is currently no API for passing in a fd to be used as QEMU logging fd. But I'm thinking about having one. It wouldn't be too hard. The other class is configurations that don't have libvirt. We need some sort of mechanism to handle QEMU logs. My intent of this email is mainly for this class of configurations. Thanks for your reply. It is very helpful. Wei.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 07/06/16 16:57, Wei Liu wrote:
I must admit I'm not familiar with the division of responsibility for managing QEMU between the Xen provided libxl library(s) and the libvirt libxl driver code. Naively I would expect the libvirt libxl driver code to deal with virtlogd and then configure the Xen libxl library / QEMU accordingly. Your request seems to imply that you will need the Xen libxl library to directly talk to virtlogd instead.
Is there any way in which it would be practical for the libvirt libxl driver to talk to virtlogd to acquire the file descriptors to use and pass those file descriptors down to the libxl library ?
There are two classes of configurations.
For libvirt + libxl, There is currently no API for passing in a fd to be used as QEMU logging fd. But I'm thinking about having one. It wouldn't be too hard.
The other class is configurations that don't have libvirt. We need some sort of mechanism to handle QEMU logs. My intent of this email is mainly for this class of configurations.
From a global perspective, #4 would seem to make sense, since it allows
Just to be clear -- internally we're investigating options for dealing with the "qemu logging" problem* for XenProject for people not running libvirt -- people who use the xl toolstack, or people who build their own toolstack on top of libxl. (We *also* need to figure out how to deal with the libxl+libvirt situation, but that's just a matter of plumbing I think.) The options we've come up with, broadly, are as follows: 1. Try to use the existing syslog facilities 2. Re-purpose one of our existing daemons to perform a role similar to virtlogd 3. "Steal" virtlogd and import it into our tree (yay GPL!) 4. Work with the libvirt community to make virtlogd an independent project which can be used by both libvirt and libxl directly None of the options are really that great. I'm sure you guys explored #1 yourselves before deciding to write your own tools, so I won't cover its deficiencies. #2 and #3 both involve a lot of duplicate effort and duplicate code. the virtlogd functionality to be more generally used (and potentially may be useful in non-virtualization scenarios as well). But it also has the cost of working cross-community, maintaining a clean separate codebase, &c &c. And we realize for the libvirt project it's extra work for no obvious immediate benefit. As Wei said, we're still exploring options; even a negative response narrows down the search space. :-) Let us know what you think. -George * For those not familiar: The challenge is to log debug messages from qemu to a file, so that if there are problems it's easier to diagnose; *without* allowing a rogue guest to fill up your disk by causing qemu to spew useless messages.
On Wed, Jun 08, 2016 at 10:50:24AM +0100, George Dunlap wrote:
On 07/06/16 16:57, Wei Liu wrote:
I must admit I'm not familiar with the division of responsibility for managing QEMU between the Xen provided libxl library(s) and the libvirt libxl driver code. Naively I would expect the libvirt libxl driver code to deal with virtlogd and then configure the Xen libxl library / QEMU accordingly. Your request seems to imply that you will need the Xen libxl library to directly talk to virtlogd instead.
Is there any way in which it would be practical for the libvirt libxl driver to talk to virtlogd to acquire the file descriptors to use and pass those file descriptors down to the libxl library ?
There are two classes of configurations.
For libvirt + libxl, There is currently no API for passing in a fd to be used as QEMU logging fd. But I'm thinking about having one. It wouldn't be too hard.
The other class is configurations that don't have libvirt. We need some sort of mechanism to handle QEMU logs. My intent of this email is mainly for this class of configurations.
Just to be clear -- internally we're investigating options for dealing with the "qemu logging" problem* for XenProject for people not running libvirt -- people who use the xl toolstack, or people who build their own toolstack on top of libxl.
(We *also* need to figure out how to deal with the libxl+libvirt situation, but that's just a matter of plumbing I think.)
The options we've come up with, broadly, are as follows:
1. Try to use the existing syslog facilities
2. Re-purpose one of our existing daemons to perform a role similar to virtlogd
3. "Steal" virtlogd and import it into our tree (yay GPL!)
4. Work with the libvirt community to make virtlogd an independent project which can be used by both libvirt and libxl directly
For completeness I'd also suggest 5. Declare it out of scope for xl toolstack to solve the whole problem. Merely provide the minimal hooks to enable the layer above libxl to solve it. This is effectively QEMU's approach. Of course, this would mean that any non-libvirt layer using libxl stil faces the same problem you're facing, so I understand if thats not desirable from your POV.
None of the options are really that great. I'm sure you guys explored #1 yourselves before deciding to write your own tools, so I won't cover its deficiencies. #2 and #3 both involve a lot of duplicate effort and duplicate code.
From a global perspective, #4 would seem to make sense, since it allows the virtlogd functionality to be more generally used (and potentially may be useful in non-virtualization scenarios as well). But it also has the cost of working cross-community, maintaining a clean separate codebase, &c &c. And we realize for the libvirt project it's extra work for no obvious immediate benefit.
As you say there's not really any pre-existing tools that can easily fit with the requirements, which is why we ended up creating virtlogd ourselves - it was either that or OpenStack was going to invent their own daemon which does what virtlogd does to solve it at the even higher layer (though they could only fix serial port file based output, not stderr outout) So, we wanted a standard solution that libvirt and all apps using libvirt could rely up unconditionally. From our existing libvirtd, and codebase in general, we have alot of infrastructure pieces that made creating virtlogd a pretty easy task. In particular our formal RPC protocol and handling code for libvirtd and virtlockd, was able to serve as the basis for virtlogd with little need for extra code. This in turn means that having virtlogd as a separate project would be a major undertaking - it relies on so much libvirt infrastructure code that to make it into a separate project, we'd have to pull out a huge pile of libvirt internal code and turn it into a more formal library that could be shared between an external virtlogd and libvirt. We've never considered any of this code to be API/ABI stable, so don't really want to go down that route. This would also make your option #3 a surprisingly large effort - there's a load of libvirt code you would have to pull into Xen, or alternatively re-write in a Xen friendly manner. Less problematic, though still relevant, is that virtlogd is intended to align with libvirtd/virtlockd designs, to give a consistent model. By this I mean the config files are in common locations, the way we auto-spawn the daemons works the same way - eg we have one libvirtd running privileged, and one per-user account, and auto-spawn a corresponding virtlogd for each.
As Wei said, we're still exploring options; even a negative response narrows down the search space. :-)
Let us know what you think.
I don't have a great answer for you I'm afraid, but I don't think that #4 is really practical from the libvirt POV, due to the issue with the all the libvirt internal code virtlogd relies upon that we don't wish to turn into a stable API/ABI. FWIW, for #3 you would have to take a large portion of src/util and src/rpc and src/logging from the libvirt tree, which is quite a considerable amount of code, so also not great.
* For those not familiar: The challenge is to log debug messages from qemu to a file, so that if there are problems it's easier to diagnose; *without* allowing a rogue guest to fill up your disk by causing qemu to spew useless messages.
NB, it is both messages from QEMU's stdout/stderr, and usage of character devices (ie seriall ports) with the 'file' backend which lets QEMU write guest serial output straight to a file. Fixing the former is possible with any QEM, while the latter requires QEMU >= 2.6 Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, Jun 08, 2016 at 11:07:16AM +0100, Daniel P. Berrange wrote: [...]
situation, but that's just a matter of plumbing I think.)
The options we've come up with, broadly, are as follows:
1. Try to use the existing syslog facilities
2. Re-purpose one of our existing daemons to perform a role similar to virtlogd
3. "Steal" virtlogd and import it into our tree (yay GPL!)
4. Work with the libvirt community to make virtlogd an independent project which can be used by both libvirt and libxl directly
For completeness I'd also suggest
5. Declare it out of scope for xl toolstack to solve the whole problem. Merely provide the minimal hooks to enable the layer above libxl to solve it. This is effectively QEMU's approach.
Of course, this would mean that any non-libvirt layer using libxl stil faces the same problem you're facing, so I understand if thats not desirable from your POV.
None of the options are really that great. I'm sure you guys explored #1 yourselves before deciding to write your own tools, so I won't cover its deficiencies. #2 and #3 both involve a lot of duplicate effort and duplicate code.
From a global perspective, #4 would seem to make sense, since it allows the virtlogd functionality to be more generally used (and potentially may be useful in non-virtualization scenarios as well). But it also has the cost of working cross-community, maintaining a clean separate codebase, &c &c. And we realize for the libvirt project it's extra work for no obvious immediate benefit.
As you say there's not really any pre-existing tools that can easily fit with the requirements, which is why we ended up creating virtlogd ourselves - it was either that or OpenStack was going to invent their own daemon which does what virtlogd does to solve it at the even higher layer (though they could only fix serial port file based output, not stderr outout)
So, we wanted a standard solution that libvirt and all apps using libvirt could rely up unconditionally. From our existing libvirtd, and codebase in general, we have alot of infrastructure pieces that made creating virtlogd a pretty easy task. In particular our formal RPC protocol and handling code for libvirtd and virtlockd, was able to serve as the basis for virtlogd with little need for extra code.
This in turn means that having virtlogd as a separate project would be a major undertaking - it relies on so much libvirt infrastructure code that to make it into a separate project, we'd have to pull out a huge pile of libvirt internal code and turn it into a more formal library that could be shared between an external virtlogd and libvirt. We've never considered any of this code to be API/ABI stable, so don't really want to go down that route. This would also make your option #3 a surprisingly large effort - there's a load of libvirt code you would have to pull into Xen, or alternatively re-write in a Xen friendly manner.
Less problematic, though still relevant, is that virtlogd is intended to align with libvirtd/virtlockd designs, to give a consistent model. By this I mean the config files are in common locations, the way we auto-spawn the daemons works the same way - eg we have one libvirtd running privileged, and one per-user account, and auto-spawn a corresponding virtlogd for each.
FWIW I came to the same conclusion in my own research. So I'm not really keen on #3 and #4.
As Wei said, we're still exploring options; even a negative response narrows down the search space. :-)
Let us know what you think.
I don't have a great answer for you I'm afraid, but I don't think that #4 is really practical from the libvirt POV, due to the issue with the all the libvirt internal code virtlogd relies upon that we don't wish to turn into a stable API/ABI.
Understood. Wei.
participants (3)
-
Daniel P. Berrange -
George Dunlap -
Wei Liu