[libvirt] [RFC] Design executing commands from within domains
Dear list, while wiring qemu-ga into libvirt I've noticed that it has ability to spawn commands inside guest. I haven't paid much attention to it then as implementing libvirt <-> qemu-ga communication was more important. But lately couple of requests on the list showed up where ability to spawn various commands inside guests would be much appreciated (e.g. when fetching some stats that HV can't know or has no support for yet - free/df/..). When it comes to spawning commands we distinguish two modes: sync and async. I think we should stick to this on the public API level too. Yet better, we can have async APIs and the sync API would just be a wrapper around async then. One picture instead of thousand words: /* obtain conn and dom somehow */ const char argv[] = {"argv1", "argv2", "argv3", ...} size_t argc = ARRAY_CARDINALITY(argv); unsigned int flags = 0; virDomainCommandPtr cmd = virDomainCommandNew("/path/to/binary", argv, argc, flags); /* Should we care about env too? */ virStreamPtr st = virSreamNew(conn, VIR_STREAM_NONBLOCK); virDomainCommandRun(dom, cmd, st, VIR_COMMAND_STRING_IO); virStreamEventAddCallback(st, VIR_STREAM_EVENT_READABLE, domainCommandCallback, ...); /* ... */ virDomainCommandJoin(dom, cmd); virStreamEventRemoveCallback(con->st) virDomainCommandFree(cmd); While qemu-ga can handle input for a binary its executing, this must be put right into JSON when constructing the qemu-ga command. It's not common that by that time users know the input they want to enter. So I have to figure out something. Opinions? Michal
On Mon, Aug 08, 2016 at 05:00:38PM +0200, Michal Privoznik wrote:
Dear list,
while wiring qemu-ga into libvirt I've noticed that it has ability to spawn commands inside guest. I haven't paid much attention to it then as implementing libvirt <-> qemu-ga communication was more important. But lately couple of requests on the list showed up where ability to spawn various commands inside guests would be much appreciated (e.g. when fetching some stats that HV can't know or has no support for yet - free/df/..).
When it comes to spawning commands we distinguish two modes: sync and async. I think we should stick to this on the public API level too. Yet better, we can have async APIs and the sync API would just be a wrapper around async then.
One picture instead of thousand words:
/* obtain conn and dom somehow */ const char argv[] = {"argv1", "argv2", "argv3", ...} size_t argc = ARRAY_CARDINALITY(argv); unsigned int flags = 0;
virDomainCommandPtr cmd = virDomainCommandNew("/path/to/binary", argv, argc, flags); /* Should we care about env too? */
virStreamPtr st = virSreamNew(conn, VIR_STREAM_NONBLOCK);
virDomainCommandRun(dom, cmd, st, VIR_COMMAND_STRING_IO);
virStreamEventAddCallback(st, VIR_STREAM_EVENT_READABLE, domainCommandCallback, ...);
/* ... */
virDomainCommandJoin(dom, cmd);
virStreamEventRemoveCallback(con->st)
virDomainCommandFree(cmd);
While qemu-ga can handle input for a binary its executing, this must be put right into JSON when constructing the qemu-ga command. It's not common that by that time users know the input they want to enter. So I have to figure out something.
Opinions?
I never really liked the qemu guest agent ability to run arbitrary commands. It is basically re-inventing the shell but with really awful features. eg the having to provide all the input upfront, not having any way to stream large stdout/stderr data back to the host. Further, from a security POV it is really bad practice to have this feature in QEMU guest agent, as it makes it impossible to provide any kind of sane security confinment for the GA. IIRC, default Fedora SELinux policy will not even permit the exec command to be to used. Most of the QEMU GA commands have very tight scope so are easily confined, but 'exec' by its very nature wants todo anything. From that POV, a general purpose exec facility is really better suited to a separate command. Also from an API modelling POV, exposing the guest agent exec in libvirt is pretty much giving up on any sense of API design. It'll just discourage anyone from ever writing any further special case guest agent commands with formal APIs. IOW, I don't think we should ever expose the qemu guest agent exec command via libvirt APIs. If people want a general purpose exec facility, they can trivially write a much better exec feature via a separate virtio-serial channel (or via the new virtio-vsock), and stil access that over libvirt by using the virDomainOpenChannel to connect to the host side of the channel if needed Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, 2016-08-08 at 16:19 +0100, Daniel P. Berrange wrote:
IOW, I don't think we should ever expose the qemu guest agent exec command via libvirt APIs. If people want a general purpose exec facility, they can trivially write a much better exec feature via a separate virtio-serial channel (or via the new virtio-vsock), and stil access that over libvirt by using the virDomainOpenChannel to connect to the host side of the channel if needed
Personally, I feel like most of this is way out of scope for libvirt. If you want to monitor the disk / memory usage of your guest, just run the Nagios agent or whatever in it; if you want to run random commands, just use ssh. These are not some specific requirements that are strongly tied to virtualization, but very generic requirements that can be addressed with existing generic software. -- Andrea Bolognani / Red Hat / Virtualization
在 2016-08-09 19:03:32，"Andrea Bolognani" <abologna@redhat.com> 写道：
On Mon, 2016-08-08 at 16:19 +0100, Daniel P. Berrange wrote:
IOW, I don't think we should ever expose the qemu guest agent exec command via libvirt APIs. If people want a general purpose exec facility, they can trivially write a much better exec feature via a separate virtio-serial channel (or via the new virtio-vsock), and stil access that over libvirt by using the virDomainOpenChannel to connect to the host side of the channel if needed
Personally, I feel like most of this is way out of scope for libvirt.
If you want to monitor the disk / memory usage of your guest, just run the Nagios agent or whatever in it; if you want to run random commands, just use ssh.
One of the use cases for guest agent exec is that when network service is abnormal, we could restart it in VMs rather than vnc/spice :) Regards, - Chen
These are not some specific requirements that are strongly tied to virtualization, but very generic requirements that can be addressed with existing generic software.
On Wed, Aug 10, 2016 at 02:32:47PM +0800, Chen Hanxiao wrote:
在 2016-08-09 19:03:32，"Andrea Bolognani" <abologna@redhat.com> 写道：
On Mon, 2016-08-08 at 16:19 +0100, Daniel P. Berrange wrote:
IOW, I don't think we should ever expose the qemu guest agent exec command via libvirt APIs. If people want a general purpose exec facility, they can trivially write a much better exec feature via a separate virtio-serial channel (or via the new virtio-vsock), and stil access that over libvirt by using the virDomainOpenChannel to connect to the host side of the channel if needed
Personally, I feel like most of this is way out of scope for libvirt.
If you want to monitor the disk / memory usage of your guest, just run the Nagios agent or whatever in it; if you want to run random commands, just use ssh.
One of the use cases for guest agent exec is that when network service is abnormal, we could restart it in VMs rather than vnc/spice :)
So just make your monitoring app be able to run over virtio-serial or virtio-vsock, to it can avoid relying on network. These are explicitly designed to allow multiple services, so we don't need to shove everything via the QEMU guest agent. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
At 2016-08-08 23:19:26, "Daniel P. Berrange" <berrange@redhat.com> wrote:
On Mon, Aug 08, 2016 at 05:00:38PM +0200, Michal Privoznik wrote:
Dear list,
while wiring qemu-ga into libvirt I've noticed that it has ability to spawn commands inside guest. I haven't paid much attention to it then as implementing libvirt <-> qemu-ga communication was more important. But lately couple of requests on the list showed up where ability to spawn various commands inside guests would be much appreciated (e.g. when fetching some stats that HV can't know or has no support for yet - free/df/..).
I never really liked the qemu guest agent ability to run arbitrary commands. It is basically re-inventing the shell but with really awful features. eg the having to provide all the input upfront, not having any way to stream large stdout/stderr data back to the host.
Further, from a security POV it is really bad practice to have this feature in QEMU guest agent, as it makes it impossible to provide any kind of sane security confinment for the GA. IIRC, default Fedora SELinux policy will not even permit the exec command to be to used. Most of the QEMU GA commands have very tight scope so are easily confined, but 'exec' by its very nature wants todo anything. From that POV, a general purpose exec facility is really better suited to a separate command.
Users could ban it inside VM by blacklist. Also, with default SELinux policy, qga could not do anything more. So it's under control.
Also from an API modelling POV, exposing the guest agent exec in libvirt is pretty much giving up on any sense of API design. It'll just discourage anyone from ever writing any further special case guest agent commands with formal APIs.
IOW, I don't think we should ever expose the qemu guest agent exec command via libvirt APIs.
We've had qemuAgentCommand. So I think it's better for a new public API. Regards, - Chen
On Tue, Aug 09, 2016 at 05:57:44PM +0800, Chen Hanxiao wrote:
At 2016-08-08 23:19:26, "Daniel P. Berrange" <berrange@redhat.com> wrote:
On Mon, Aug 08, 2016 at 05:00:38PM +0200, Michal Privoznik wrote:
Dear list,
while wiring qemu-ga into libvirt I've noticed that it has ability to spawn commands inside guest. I haven't paid much attention to it then as implementing libvirt <-> qemu-ga communication was more important. But lately couple of requests on the list showed up where ability to spawn various commands inside guests would be much appreciated (e.g. when fetching some stats that HV can't know or has no support for yet - free/df/..).
I never really liked the qemu guest agent ability to run arbitrary commands. It is basically re-inventing the shell but with really awful features. eg the having to provide all the input upfront, not having any way to stream large stdout/stderr data back to the host.
Further, from a security POV it is really bad practice to have this feature in QEMU guest agent, as it makes it impossible to provide any kind of sane security confinment for the GA. IIRC, default Fedora SELinux policy will not even permit the exec command to be to used. Most of the QEMU GA commands have very tight scope so are easily confined, but 'exec' by its very nature wants todo anything. From that POV, a general purpose exec facility is really better suited to a separate command.
Users could ban it inside VM by blacklist. Also, with default SELinux policy, qga could not do anything more. So it's under control.
No, it really isn't satisfactory, because it does not enable you to separate responsibilities. ie the set of clients you want to grant 'exec' access to, is not the same as the set of clients you want to give general QEMU guest agent access to. By serving both use cases with the same agent, once you allow 'exec' access for one use cases, you've not opened that avenue of attack to all users of qemu guest agent. If a general purpose 'exec' feature is desired, it can be achieved by doing something as simple as running mingetty on a new well-defined virtio-serial port. eg org.qemu.exec, and configuring that with PAM to not require a password login. That gives the client app full shell job control, I/O streaming, etc, ie all the things that would be broken/impossible with trying to wrap the qemu guest agent exec feature into an API.
Also from an API modelling POV, exposing the guest agent exec in libvirt is pretty much giving up on any sense of API design. It'll just discourage anyone from ever writing any further special case guest agent commands with formal APIs.
IOW, I don't think we should ever expose the qemu guest agent exec command via libvirt APIs.
We've had qemuAgentCommand. So I think it's better for a new public API.
Thats in the libvirt-qemu.h file, so not something which has the same long term support guarantee as formal public APIs. So existance of that API is not justification to add a general guest exec to the main libvirt API. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
At 2016-08-09 19:02:18, "Daniel P. Berrange" <berrange@redhat.com> wrote:
On Tue, Aug 09, 2016 at 05:57:44PM +0800, Chen Hanxiao wrote:
At 2016-08-08 23:19:26, "Daniel P. Berrange" <berrange@redhat.com> wrote:
On Mon, Aug 08, 2016 at 05:00:38PM +0200, Michal Privoznik wrote:
Dear list,
while wiring qemu-ga into libvirt I've noticed that it has ability to spawn commands inside guest. I haven't paid much attention to it then as implementing libvirt <-> qemu-ga communication was more important. But lately couple of requests on the list showed up where ability to spawn various commands inside guests would be much appreciated (e.g. when fetching some stats that HV can't know or has no support for yet - free/df/..).
I never really liked the qemu guest agent ability to run arbitrary commands. It is basically re-inventing the shell but with really awful features. eg the having to provide all the input upfront, not having any way to stream large stdout/stderr data back to the host.
Further, from a security POV it is really bad practice to have this feature in QEMU guest agent, as it makes it impossible to provide any kind of sane security confinment for the GA. IIRC, default Fedora SELinux policy will not even permit the exec command to be to used. Most of the QEMU GA commands have very tight scope so are easily confined, but 'exec' by its very nature wants todo anything. From that POV, a general purpose exec facility is really better suited to a separate command.
Users could ban it inside VM by blacklist. Also, with default SELinux policy, qga could not do anything more. So it's under control.
No, it really isn't satisfactory, because it does not enable you to separate responsibilities. ie the set of clients you want to grant 'exec' access to, is not the same as the set of clients you want to give general QEMU guest agent access to. By serving both use cases with the same agent, once you allow 'exec' access for one use cases, you've not opened that avenue of attack to all users of qemu guest agent.
That's a problem. Although end user of VM could protect themselves by disable guest agent or by --blacklist as RHEL did.
If a general purpose 'exec' feature is desired, it can be achieved by doing something as simple as running mingetty on a new well-defined virtio-serial port. eg org.qemu.exec, and configuring that with PAM to not require a password login. That gives the client app full shell job control, I/O streaming, etc, ie all the things that would be broken/impossible with trying to wrap the qemu guest agent exec feature into an API.
Also from an API modelling POV, exposing the guest agent exec in libvirt is pretty much giving up on any sense of API design. It'll just discourage anyone from ever writing any further special case guest agent commands with formal APIs.
IOW, I don't think we should ever expose the qemu guest agent exec command via libvirt APIs.
We've had qemuAgentCommand. So I think it's better for a new public API.
Thats in the libvirt-qemu.h file, so not something which has the same long term support guarantee as formal public APIs. So existance of that API is not justification to add a general guest exec to the main libvirt API.
Thanks for your clarification. Regards, - Chen
At 2016-08-08 23:00:38, "Michal Privoznik" <mprivozn@redhat.com> wrote:
Dear list,
while wiring qemu-ga into libvirt I've noticed that it has ability to spawn commands inside guest. I haven't paid much attention to it then as implementing libvirt <-> qemu-ga communication was more important. But lately couple of requests on the list showed up where ability to spawn various commands inside guests would be much appreciated (e.g. when fetching some stats that HV can't know or has no support for yet - free/df/..).
When it comes to spawning commands we distinguish two modes: sync and async. I think we should stick to this on the public API level too. Yet better, we can have async APIs and the sync API would just be a wrapper around async then.
Cool! We should had a API than using qemu monitor command. ...
virStreamEventAddCallback(st, VIR_STREAM_EVENT_READABLE, domainCommandCallback, ...);
/* ... */
virDomainCommandJoin(dom, cmd);
virStreamEventRemoveCallback(con->st)
virDomainCommandFree(cmd);
Do we had timeout mechanism? Some command may hang for a long time in guest. Regards, - Chen
While qemu-ga can handle input for a binary its executing, this must be put right into JSON when constructing the qemu-ga command. It's not common that by that time users know the input they want to enter. So I have to figure out something.
Opinions?
Michal
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Tue, Aug 09, 2016 at 05:48:45PM +0800, Chen Hanxiao wrote:
At 2016-08-08 23:00:38, "Michal Privoznik" <mprivozn@redhat.com> wrote:
Dear list,
while wiring qemu-ga into libvirt I've noticed that it has ability to spawn commands inside guest. I haven't paid much attention to it then as implementing libvirt <-> qemu-ga communication was more important. But lately couple of requests on the list showed up where ability to spawn various commands inside guests would be much appreciated (e.g. when fetching some stats that HV can't know or has no support for yet - free/df/..).
When it comes to spawning commands we distinguish two modes: sync and async. I think we should stick to this on the public API level too. Yet better, we can have async APIs and the sync API would just be a wrapper around async then.
Cool! We should had a API than using qemu monitor command.
...
virStreamEventAddCallback(st, VIR_STREAM_EVENT_READABLE, domainCommandCallback, ...);
/* ... */
virDomainCommandJoin(dom, cmd);
virStreamEventRemoveCallback(con->st)
virDomainCommandFree(cmd);
Do we had timeout mechanism? Some command may hang for a long time in guest.
Yet another reason to not try to re-invent a mechanism for running shell commands in libvirt API, and just run a shell service over a separate virtio-serial port instead. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (4)
-
Andrea Bolognani -
Chen Hanxiao -
Daniel P. Berrange -
Michal Privoznik