Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)
On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: 2. Ability to watch QMP activity on a running QEMU process, e.g. even when libvirt is directly connected to the monitor.
That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy works.
(Actually, mitmproxy could theoretically be taught how to read and understand QMP traffic, but that's not something I know how to do or would be prepared to mentor.)
Is this possible to do in a post-hoc fashion? Let's say you are using production environment QEMU, how do we attach the QMP listener to it? Or does this idea require that we start QEMU in a specific fashion with a second debug socket that qmp-shell can connect to in order to listen?
... Or do we engineer qmp-shell to open its own socket that libvirt connects to ...?
Here is the QEMU command-line that libvirt uses on my F33 system: -chardev socket,id=charmonitor,fd=36,server,nowait -mon chardev=charmonitor,id=monitor,mode=control Goals for this feature: 1. No manual steps required for setup. 2. Ability to start/stop monitoring traffic at runtime without restarting QEMU. 3. Available to unprivileged users. I think the easiest way to achieve this is through a new QEMU monitor command. Approaches that come to mind: 1. Add a -mon debug-chardev property and a QMP command to set it at runtime. The debug-chardev receives both monitor input (commands) and output (responses and events). This does not allow MITM, rather it mirrors traffic. 2. Add a chardev-get-fd command that fetches the fd from a chardev and then use the existing chardev-change command to replace the monitor chardev with a chardev connected to qmp-shell. This inserts qmp-shell as a proxy between the QMP client and server. qmp-shell can remove itself again with another chardev-change command. This approach allows MITM. The downside is it assumes the QMP chardev is a file descriptor, so it won't work with all types of chardev. 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin source chardev, 2. a monitoring sink chardev, and 3. a monitoring source chardev. The data flow is origin <-> monitoring sink <-> monitoring source <-> QMP monitor. qmp-shell creates the monitoring sink (for receiving incoming QMP commands) and monitoring source chardev (for forwarding QMP commands or MITM commands), and then it uses change-chardev to instantiate a chardev-proxy that directs the original libvirt chardev through the monitoring sink and source. This is the most complex but also completely contained within the QEMU chardev layer. In all these approaches qmp-shell uses virsh qemu-monitor-command or an equivalent API to start/stop monitoring a running VM without manual setup steps. Stefan
On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: 2. Ability to watch QMP activity on a running QEMU process, e.g. even when libvirt is directly connected to the monitor.
That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy works.
(Actually, mitmproxy could theoretically be taught how to read and understand QMP traffic, but that's not something I know how to do or would be prepared to mentor.)
Is this possible to do in a post-hoc fashion? Let's say you are using production environment QEMU, how do we attach the QMP listener to it? Or does this idea require that we start QEMU in a specific fashion with a second debug socket that qmp-shell can connect to in order to listen?
... Or do we engineer qmp-shell to open its own socket that libvirt connects to ...?
Here is the QEMU command-line that libvirt uses on my F33 system:
-chardev socket,id=charmonitor,fd=36,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
Goals for this feature:
1. No manual steps required for setup. 2. Ability to start/stop monitoring traffic at runtime without restarting QEMU. 3. Available to unprivileged users.
I think the easiest way to achieve this is through a new QEMU monitor command. Approaches that come to mind:
1. Add a -mon debug-chardev property and a QMP command to set it at runtime. The debug-chardev receives both monitor input (commands) and output (responses and events). This does not allow MITM, rather it mirrors traffic.
2. Add a chardev-get-fd command that fetches the fd from a chardev and then use the existing chardev-change command to replace the monitor chardev with a chardev connected to qmp-shell. This inserts qmp-shell as a proxy between the QMP client and server. qmp-shell can remove itself again with another chardev-change command. This approach allows MITM. The downside is it assumes the QMP chardev is a file descriptor, so it won't work with all types of chardev.
3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin source chardev, 2. a monitoring sink chardev, and 3. a monitoring source chardev. The data flow is origin <-> monitoring sink <-> monitoring source <-> QMP monitor. qmp-shell creates the monitoring sink (for receiving incoming QMP commands) and monitoring source chardev (for forwarding QMP commands or MITM commands), and then it uses change-chardev to instantiate a chardev-proxy that directs the original libvirt chardev through the monitoring sink and source.
This is the most complex but also completely contained within the QEMU chardev layer.
In all these approaches qmp-shell uses virsh qemu-monitor-command or an equivalent API to start/stop monitoring a running VM without manual setup steps.
Why go to the trouble of adding more chardevs to a running QEMU that libvirt has. qmp-shell can just directly use the libvirt Python API to invoke virDomainQemuMonitorCommand to invoke QMP commands, and the othe API for receiving QMP events. Essentially it just needs to be split into two layers. The upper layer works in terms of individual QMP command/replies, and QMP events. The lower layer provides a transport that is either a UNIX socket, or is the libvirt QMP passthrough API. Or alternatively, provide a virt-qmp-shim command that listens on a UNIX socket, accepts QMP commands and turns them into calls to virDomainQemuMonitorCommand, and funnells back the response. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: 2. Ability to watch QMP activity on a running QEMU process, e.g. even when libvirt is directly connected to the monitor.
That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy works.
(Actually, mitmproxy could theoretically be taught how to read and understand QMP traffic, but that's not something I know how to do or would be prepared to mentor.)
Is this possible to do in a post-hoc fashion? Let's say you are using production environment QEMU, how do we attach the QMP listener to it? Or does this idea require that we start QEMU in a specific fashion with a second debug socket that qmp-shell can connect to in order to listen?
... Or do we engineer qmp-shell to open its own socket that libvirt connects to ...?
Here is the QEMU command-line that libvirt uses on my F33 system:
-chardev socket,id=charmonitor,fd=36,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
Goals for this feature:
1. No manual steps required for setup. 2. Ability to start/stop monitoring traffic at runtime without restarting QEMU. 3. Available to unprivileged users.
I think the easiest way to achieve this is through a new QEMU monitor command. Approaches that come to mind:
1. Add a -mon debug-chardev property and a QMP command to set it at runtime. The debug-chardev receives both monitor input (commands) and output (responses and events). This does not allow MITM, rather it mirrors traffic.
2. Add a chardev-get-fd command that fetches the fd from a chardev and then use the existing chardev-change command to replace the monitor chardev with a chardev connected to qmp-shell. This inserts qmp-shell as a proxy between the QMP client and server. qmp-shell can remove itself again with another chardev-change command. This approach allows MITM. The downside is it assumes the QMP chardev is a file descriptor, so it won't work with all types of chardev.
3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin source chardev, 2. a monitoring sink chardev, and 3. a monitoring source chardev. The data flow is origin <-> monitoring sink <-> monitoring source <-> QMP monitor. qmp-shell creates the monitoring sink (for receiving incoming QMP commands) and monitoring source chardev (for forwarding QMP commands or MITM commands), and then it uses change-chardev to instantiate a chardev-proxy that directs the original libvirt chardev through the monitoring sink and source.
This is the most complex but also completely contained within the QEMU chardev layer.
I have an idea for the QMP command name: chardev-snapshot-sync! Finally we get backing file chains for chardevs! :-)
In all these approaches qmp-shell uses virsh qemu-monitor-command or an equivalent API to start/stop monitoring a running VM without manual setup steps.
Why go to the trouble of adding more chardevs to a running QEMU that libvirt has. qmp-shell can just directly use the libvirt Python API to invoke virDomainQemuMonitorCommand to invoke QMP commands, and the othe API for receiving QMP events.
Essentially it just needs to be split into two layers. The upper layer works in terms of individual QMP command/replies, and QMP events. The lower layer provides a transport that is either a UNIX socket, or is the libvirt QMP passthrough API.
Or alternatively, provide a virt-qmp-shim command that listens on a UNIX socket, accepts QMP commands and turns them into calls to virDomainQemuMonitorCommand, and funnells back the response.
I think the idea was to show the QMP traffic that libvirt produces for other management applications, not for the QMP shell. These APIs probably don't allow this? Kevin
On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: 2. Ability to watch QMP activity on a running QEMU process, e.g. even when libvirt is directly connected to the monitor.
That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy works.
(Actually, mitmproxy could theoretically be taught how to read and understand QMP traffic, but that's not something I know how to do or would be prepared to mentor.)
Is this possible to do in a post-hoc fashion? Let's say you are using production environment QEMU, how do we attach the QMP listener to it? Or does this idea require that we start QEMU in a specific fashion with a second debug socket that qmp-shell can connect to in order to listen?
... Or do we engineer qmp-shell to open its own socket that libvirt connects to ...?
Here is the QEMU command-line that libvirt uses on my F33 system:
-chardev socket,id=charmonitor,fd=36,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
Goals for this feature:
1. No manual steps required for setup. 2. Ability to start/stop monitoring traffic at runtime without restarting QEMU. 3. Available to unprivileged users.
I think the easiest way to achieve this is through a new QEMU monitor command. Approaches that come to mind:
1. Add a -mon debug-chardev property and a QMP command to set it at runtime. The debug-chardev receives both monitor input (commands) and output (responses and events). This does not allow MITM, rather it mirrors traffic.
2. Add a chardev-get-fd command that fetches the fd from a chardev and then use the existing chardev-change command to replace the monitor chardev with a chardev connected to qmp-shell. This inserts qmp-shell as a proxy between the QMP client and server. qmp-shell can remove itself again with another chardev-change command. This approach allows MITM. The downside is it assumes the QMP chardev is a file descriptor, so it won't work with all types of chardev.
3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin source chardev, 2. a monitoring sink chardev, and 3. a monitoring source chardev. The data flow is origin <-> monitoring sink <-> monitoring source <-> QMP monitor. qmp-shell creates the monitoring sink (for receiving incoming QMP commands) and monitoring source chardev (for forwarding QMP commands or MITM commands), and then it uses change-chardev to instantiate a chardev-proxy that directs the original libvirt chardev through the monitoring sink and source.
This is the most complex but also completely contained within the QEMU chardev layer.
I have an idea for the QMP command name: chardev-snapshot-sync!
Finally we get backing file chains for chardevs! :-)
In all these approaches qmp-shell uses virsh qemu-monitor-command or an equivalent API to start/stop monitoring a running VM without manual setup steps.
Why go to the trouble of adding more chardevs to a running QEMU that libvirt has. qmp-shell can just directly use the libvirt Python API to invoke virDomainQemuMonitorCommand to invoke QMP commands, and the othe API for receiving QMP events.
Essentially it just needs to be split into two layers. The upper layer works in terms of individual QMP command/replies, and QMP events. The lower layer provides a transport that is either a UNIX socket, or is the libvirt QMP passthrough API.
Or alternatively, provide a virt-qmp-shim command that listens on a UNIX socket, accepts QMP commands and turns them into calls to virDomainQemuMonitorCommand, and funnells back the response.
I think the idea was to show the QMP traffic that libvirt produces for other management applications, not for the QMP shell. These APIs probably don't allow this?
FWIW if you want to monitor what libvirt is sending/receiving we have a script for that that uses our systemtap probe points: https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-mon... Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote:
On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: 2. Ability to watch QMP activity on a running QEMU process, e.g. even when libvirt is directly connected to the monitor.
That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy works.
(Actually, mitmproxy could theoretically be taught how to read and understand QMP traffic, but that's not something I know how to do or would be prepared to mentor.)
Is this possible to do in a post-hoc fashion? Let's say you are using production environment QEMU, how do we attach the QMP listener to it? Or does this idea require that we start QEMU in a specific fashion with a second debug socket that qmp-shell can connect to in order to listen?
... Or do we engineer qmp-shell to open its own socket that libvirt connects to ...?
Here is the QEMU command-line that libvirt uses on my F33 system:
-chardev socket,id=charmonitor,fd=36,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
Goals for this feature:
1. No manual steps required for setup. 2. Ability to start/stop monitoring traffic at runtime without restarting QEMU. 3. Available to unprivileged users.
I think the easiest way to achieve this is through a new QEMU monitor command. Approaches that come to mind:
1. Add a -mon debug-chardev property and a QMP command to set it at runtime. The debug-chardev receives both monitor input (commands) and output (responses and events). This does not allow MITM, rather it mirrors traffic.
2. Add a chardev-get-fd command that fetches the fd from a chardev and then use the existing chardev-change command to replace the monitor chardev with a chardev connected to qmp-shell. This inserts qmp-shell as a proxy between the QMP client and server. qmp-shell can remove itself again with another chardev-change command. This approach allows MITM. The downside is it assumes the QMP chardev is a file descriptor, so it won't work with all types of chardev.
3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin source chardev, 2. a monitoring sink chardev, and 3. a monitoring source chardev. The data flow is origin <-> monitoring sink <-> monitoring source <-> QMP monitor. qmp-shell creates the monitoring sink (for receiving incoming QMP commands) and monitoring source chardev (for forwarding QMP commands or MITM commands), and then it uses change-chardev to instantiate a chardev-proxy that directs the original libvirt chardev through the monitoring sink and source.
This is the most complex but also completely contained within the QEMU chardev layer.
I have an idea for the QMP command name: chardev-snapshot-sync!
Finally we get backing file chains for chardevs! :-)
In all these approaches qmp-shell uses virsh qemu-monitor-command or an equivalent API to start/stop monitoring a running VM without manual setup steps.
Why go to the trouble of adding more chardevs to a running QEMU that libvirt has. qmp-shell can just directly use the libvirt Python API to invoke virDomainQemuMonitorCommand to invoke QMP commands, and the othe API for receiving QMP events.
Essentially it just needs to be split into two layers. The upper layer works in terms of individual QMP command/replies, and QMP events. The lower layer provides a transport that is either a UNIX socket, or is the libvirt QMP passthrough API.
Or alternatively, provide a virt-qmp-shim command that listens on a UNIX socket, accepts QMP commands and turns them into calls to virDomainQemuMonitorCommand, and funnells back the response.
I think the idea was to show the QMP traffic that libvirt produces for other management applications, not for the QMP shell. These APIs probably don't allow this?
FWIW if you want to monitor what libvirt is sending/receiving we have a script for that that uses our systemtap probe points:
https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-mon...
Does that require root? Stefan
On Thu, Jan 14, 2021 at 04:49:17PM +0000, Stefan Hajnoczi wrote:
On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote:
On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > when libvirt is directly connected to the monitor. >
That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy works.
(Actually, mitmproxy could theoretically be taught how to read and understand QMP traffic, but that's not something I know how to do or would be prepared to mentor.)
Is this possible to do in a post-hoc fashion? Let's say you are using production environment QEMU, how do we attach the QMP listener to it? Or does this idea require that we start QEMU in a specific fashion with a second debug socket that qmp-shell can connect to in order to listen?
... Or do we engineer qmp-shell to open its own socket that libvirt connects to ...?
Here is the QEMU command-line that libvirt uses on my F33 system:
-chardev socket,id=charmonitor,fd=36,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
Goals for this feature:
1. No manual steps required for setup. 2. Ability to start/stop monitoring traffic at runtime without restarting QEMU. 3. Available to unprivileged users.
I think the easiest way to achieve this is through a new QEMU monitor command. Approaches that come to mind:
1. Add a -mon debug-chardev property and a QMP command to set it at runtime. The debug-chardev receives both monitor input (commands) and output (responses and events). This does not allow MITM, rather it mirrors traffic.
2. Add a chardev-get-fd command that fetches the fd from a chardev and then use the existing chardev-change command to replace the monitor chardev with a chardev connected to qmp-shell. This inserts qmp-shell as a proxy between the QMP client and server. qmp-shell can remove itself again with another chardev-change command. This approach allows MITM. The downside is it assumes the QMP chardev is a file descriptor, so it won't work with all types of chardev.
3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin source chardev, 2. a monitoring sink chardev, and 3. a monitoring source chardev. The data flow is origin <-> monitoring sink <-> monitoring source <-> QMP monitor. qmp-shell creates the monitoring sink (for receiving incoming QMP commands) and monitoring source chardev (for forwarding QMP commands or MITM commands), and then it uses change-chardev to instantiate a chardev-proxy that directs the original libvirt chardev through the monitoring sink and source.
This is the most complex but also completely contained within the QEMU chardev layer.
I have an idea for the QMP command name: chardev-snapshot-sync!
Finally we get backing file chains for chardevs! :-)
In all these approaches qmp-shell uses virsh qemu-monitor-command or an equivalent API to start/stop monitoring a running VM without manual setup steps.
Why go to the trouble of adding more chardevs to a running QEMU that libvirt has. qmp-shell can just directly use the libvirt Python API to invoke virDomainQemuMonitorCommand to invoke QMP commands, and the othe API for receiving QMP events.
Essentially it just needs to be split into two layers. The upper layer works in terms of individual QMP command/replies, and QMP events. The lower layer provides a transport that is either a UNIX socket, or is the libvirt QMP passthrough API.
Or alternatively, provide a virt-qmp-shim command that listens on a UNIX socket, accepts QMP commands and turns them into calls to virDomainQemuMonitorCommand, and funnells back the response.
I think the idea was to show the QMP traffic that libvirt produces for other management applications, not for the QMP shell. These APIs probably don't allow this?
FWIW if you want to monitor what libvirt is sending/receiving we have a script for that that uses our systemtap probe points:
https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-mon...
Does that require root?
Yeah, systemtap generally requires root. The same info is also written to the log files. For example: virt-admin daemon-log-filters "2:qemu_monitor_json" virt-admin daemon-lop-outputs "2:file:/var/log/libvirt/libvirtd.log" nb, i'm using level '2' there to avoid enabling debug logs, only info level logs which is the level dynamic probes log at. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Jan 14, 2021 at 04:55:30PM +0000, Daniel P. Berrangé wrote:
On Thu, Jan 14, 2021 at 04:49:17PM +0000, Stefan Hajnoczi wrote:
On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote:
On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > when libvirt is directly connected to the monitor. > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > works. > > (Actually, mitmproxy could theoretically be taught how to read and > understand QMP traffic, but that's not something I know how to do or would > be prepared to mentor.) > > Is this possible to do in a post-hoc fashion? Let's say you are using > production environment QEMU, how do we attach the QMP listener to it? Or > does this idea require that we start QEMU in a specific fashion with a > second debug socket that qmp-shell can connect to in order to listen? > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > to ...?
Here is the QEMU command-line that libvirt uses on my F33 system:
-chardev socket,id=charmonitor,fd=36,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
Goals for this feature:
1. No manual steps required for setup. 2. Ability to start/stop monitoring traffic at runtime without restarting QEMU. 3. Available to unprivileged users.
I think the easiest way to achieve this is through a new QEMU monitor command. Approaches that come to mind:
1. Add a -mon debug-chardev property and a QMP command to set it at runtime. The debug-chardev receives both monitor input (commands) and output (responses and events). This does not allow MITM, rather it mirrors traffic.
2. Add a chardev-get-fd command that fetches the fd from a chardev and then use the existing chardev-change command to replace the monitor chardev with a chardev connected to qmp-shell. This inserts qmp-shell as a proxy between the QMP client and server. qmp-shell can remove itself again with another chardev-change command. This approach allows MITM. The downside is it assumes the QMP chardev is a file descriptor, so it won't work with all types of chardev.
3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin source chardev, 2. a monitoring sink chardev, and 3. a monitoring source chardev. The data flow is origin <-> monitoring sink <-> monitoring source <-> QMP monitor. qmp-shell creates the monitoring sink (for receiving incoming QMP commands) and monitoring source chardev (for forwarding QMP commands or MITM commands), and then it uses change-chardev to instantiate a chardev-proxy that directs the original libvirt chardev through the monitoring sink and source.
This is the most complex but also completely contained within the QEMU chardev layer.
I have an idea for the QMP command name: chardev-snapshot-sync!
Finally we get backing file chains for chardevs! :-)
In all these approaches qmp-shell uses virsh qemu-monitor-command or an equivalent API to start/stop monitoring a running VM without manual setup steps.
Why go to the trouble of adding more chardevs to a running QEMU that libvirt has. qmp-shell can just directly use the libvirt Python API to invoke virDomainQemuMonitorCommand to invoke QMP commands, and the othe API for receiving QMP events.
Essentially it just needs to be split into two layers. The upper layer works in terms of individual QMP command/replies, and QMP events. The lower layer provides a transport that is either a UNIX socket, or is the libvirt QMP passthrough API.
Or alternatively, provide a virt-qmp-shim command that listens on a UNIX socket, accepts QMP commands and turns them into calls to virDomainQemuMonitorCommand, and funnells back the response.
I think the idea was to show the QMP traffic that libvirt produces for other management applications, not for the QMP shell. These APIs probably don't allow this?
FWIW if you want to monitor what libvirt is sending/receiving we have a script for that that uses our systemtap probe points:
https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-mon...
Does that require root?
Yeah, systemtap generally requires root.
The same info is also written to the log files. For example:
virt-admin daemon-log-filters "2:qemu_monitor_json" virt-admin daemon-lop-outputs "2:file:/var/log/libvirt/libvirtd.log"
nb, i'm using level '2' there to avoid enabling debug logs, only info level logs which is the level dynamic probes log at.
On my F33 system /var/log/libvirt is owned by root:root and rwx------, so I guess it would be necessary to reconfigure log output so that unprivileged users can access it. If it can be used in conjunction with virDomainQemuMonitorCommand(), then that eliminates the need to introduce new chardev functionality in QEMU. Parsing libvirt logs was one of the things I suggested, though. I think it would be a nice feature for troubleshooting QMP conversations. Stefan
On Thu, Jan 14, 2021 at 05:14:28PM +0000, Stefan Hajnoczi wrote:
On Thu, Jan 14, 2021 at 04:55:30PM +0000, Daniel P. Berrangé wrote:
On Thu, Jan 14, 2021 at 04:49:17PM +0000, Stefan Hajnoczi wrote:
On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote:
On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote: > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > when libvirt is directly connected to the monitor. > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > works. > > > > (Actually, mitmproxy could theoretically be taught how to read and > > understand QMP traffic, but that's not something I know how to do or would > > be prepared to mentor.) > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > production environment QEMU, how do we attach the QMP listener to it? Or > > does this idea require that we start QEMU in a specific fashion with a > > second debug socket that qmp-shell can connect to in order to listen? > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > > to ...? > > Here is the QEMU command-line that libvirt uses on my F33 system: > > -chardev socket,id=charmonitor,fd=36,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control > > Goals for this feature: > > 1. No manual steps required for setup. > 2. Ability to start/stop monitoring traffic at runtime without > restarting QEMU. > 3. Available to unprivileged users. > > I think the easiest way to achieve this is through a new QEMU monitor > command. Approaches that come to mind: > > 1. Add a -mon debug-chardev property and a QMP command to set it at > runtime. The debug-chardev receives both monitor input (commands) and > output (responses and events). This does not allow MITM, rather it > mirrors traffic. > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > then use the existing chardev-change command to replace the monitor > chardev with a chardev connected to qmp-shell. This inserts qmp-shell > as a proxy between the QMP client and server. qmp-shell can remove > itself again with another chardev-change command. This approach > allows MITM. The downside is it assumes the QMP chardev is a file > descriptor, so it won't work with all types of chardev. > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > source chardev, 2. a monitoring sink chardev, and 3. a monitoring > source chardev. The data flow is origin <-> monitoring sink <-> > monitoring source <-> QMP monitor. qmp-shell creates the monitoring > sink (for receiving incoming QMP commands) and monitoring source > chardev (for forwarding QMP commands or MITM commands), and then it > uses change-chardev to instantiate a chardev-proxy that directs the > original libvirt chardev through the monitoring sink and source. > > This is the most complex but also completely contained within the > QEMU chardev layer.
I have an idea for the QMP command name: chardev-snapshot-sync!
Finally we get backing file chains for chardevs! :-)
> In all these approaches qmp-shell uses virsh qemu-monitor-command or an > equivalent API to start/stop monitoring a running VM without manual > setup steps.
Why go to the trouble of adding more chardevs to a running QEMU that libvirt has. qmp-shell can just directly use the libvirt Python API to invoke virDomainQemuMonitorCommand to invoke QMP commands, and the othe API for receiving QMP events.
Essentially it just needs to be split into two layers. The upper layer works in terms of individual QMP command/replies, and QMP events. The lower layer provides a transport that is either a UNIX socket, or is the libvirt QMP passthrough API.
Or alternatively, provide a virt-qmp-shim command that listens on a UNIX socket, accepts QMP commands and turns them into calls to virDomainQemuMonitorCommand, and funnells back the response.
I think the idea was to show the QMP traffic that libvirt produces for other management applications, not for the QMP shell. These APIs probably don't allow this?
FWIW if you want to monitor what libvirt is sending/receiving we have a script for that that uses our systemtap probe points:
https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-mon...
Does that require root?
Yeah, systemtap generally requires root.
The same info is also written to the log files. For example:
virt-admin daemon-log-filters "2:qemu_monitor_json" virt-admin daemon-lop-outputs "2:file:/var/log/libvirt/libvirtd.log"
nb, i'm using level '2' there to avoid enabling debug logs, only info level logs which is the level dynamic probes log at.
On my F33 system /var/log/libvirt is owned by root:root and rwx------, so I guess it would be necessary to reconfigure log output so that unprivileged users can access it.
If you're connected to the privileged libvirtd then you have effectively got root privileges, so that shouldn't be an issue. If you're connecting to the unprivileged libvirtd, then you would tell it to use $HOME/.cache/libvirt/libvirtd.log instead.
If it can be used in conjunction with virDomainQemuMonitorCommand(), then that eliminates the need to introduce new chardev functionality in QEMU.
Parsing libvirt logs was one of the things I suggested, though. I think it would be a nice feature for troubleshooting QMP conversations.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: 2. Ability to watch QMP activity on a running QEMU process, e.g. even when libvirt is directly connected to the monitor.
That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy works.
(Actually, mitmproxy could theoretically be taught how to read and understand QMP traffic, but that's not something I know how to do or would be prepared to mentor.)
Is this possible to do in a post-hoc fashion? Let's say you are using production environment QEMU, how do we attach the QMP listener to it? Or does this idea require that we start QEMU in a specific fashion with a second debug socket that qmp-shell can connect to in order to listen?
... Or do we engineer qmp-shell to open its own socket that libvirt connects to ...?
Here is the QEMU command-line that libvirt uses on my F33 system:
-chardev socket,id=charmonitor,fd=36,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
Goals for this feature:
1. No manual steps required for setup. 2. Ability to start/stop monitoring traffic at runtime without restarting QEMU. 3. Available to unprivileged users.
I think the easiest way to achieve this is through a new QEMU monitor command. Approaches that come to mind:
1. Add a -mon debug-chardev property and a QMP command to set it at runtime. The debug-chardev receives both monitor input (commands) and output (responses and events). This does not allow MITM, rather it mirrors traffic.
2. Add a chardev-get-fd command that fetches the fd from a chardev and then use the existing chardev-change command to replace the monitor chardev with a chardev connected to qmp-shell. This inserts qmp-shell as a proxy between the QMP client and server. qmp-shell can remove itself again with another chardev-change command. This approach allows MITM. The downside is it assumes the QMP chardev is a file descriptor, so it won't work with all types of chardev.
3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin source chardev, 2. a monitoring sink chardev, and 3. a monitoring source chardev. The data flow is origin <-> monitoring sink <-> monitoring source <-> QMP monitor. qmp-shell creates the monitoring sink (for receiving incoming QMP commands) and monitoring source chardev (for forwarding QMP commands or MITM commands), and then it uses change-chardev to instantiate a chardev-proxy that directs the original libvirt chardev through the monitoring sink and source.
This is the most complex but also completely contained within the QEMU chardev layer.
I have an idea for the QMP command name: chardev-snapshot-sync!
Finally we get backing file chains for chardevs! :-)
Help mom, I'm being made fun of on qemu-devel! :)
In all these approaches qmp-shell uses virsh qemu-monitor-command or an equivalent API to start/stop monitoring a running VM without manual setup steps.
Why go to the trouble of adding more chardevs to a running QEMU that libvirt has. qmp-shell can just directly use the libvirt Python API to invoke virDomainQemuMonitorCommand to invoke QMP commands, and the othe API for receiving QMP events.
Essentially it just needs to be split into two layers. The upper layer works in terms of individual QMP command/replies, and QMP events. The lower layer provides a transport that is either a UNIX socket, or is the libvirt QMP passthrough API.
Or alternatively, provide a virt-qmp-shim command that listens on a UNIX socket, accepts QMP commands and turns them into calls to virDomainQemuMonitorCommand, and funnells back the response.
I think the idea was to show the QMP traffic that libvirt produces for other management applications, not for the QMP shell. These APIs probably don't allow this?
Yes. Stefan
On 1/14/21 8:52 AM, Stefan Hajnoczi wrote:
On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: 2. Ability to watch QMP activity on a running QEMU process, e.g. even when libvirt is directly connected to the monitor.
That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy works.
(Actually, mitmproxy could theoretically be taught how to read and understand QMP traffic, but that's not something I know how to do or would be prepared to mentor.)
Is this possible to do in a post-hoc fashion? Let's say you are using production environment QEMU, how do we attach the QMP listener to it? Or does this idea require that we start QEMU in a specific fashion with a second debug socket that qmp-shell can connect to in order to listen?
... Or do we engineer qmp-shell to open its own socket that libvirt connects to ...?
Here is the QEMU command-line that libvirt uses on my F33 system:
-chardev socket,id=charmonitor,fd=36,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
Goals for this feature:
1. No manual steps required for setup. 2. Ability to start/stop monitoring traffic at runtime without restarting QEMU. 3. Available to unprivileged users.
Excellent goals, and I agree completely.
I think the easiest way to achieve this is through a new QEMU monitor command. Approaches that come to mind:
1. Add a -mon debug-chardev property and a QMP command to set it at runtime. The debug-chardev receives both monitor input (commands) and output (responses and events). This does not allow MITM, rather it mirrors traffic.
So you have a socket that relays I/O. I wonder if it needs to modify the stream format to some extent to annotate directionality? For now, directionality can be inferred, but maybe that's brittle. (greeting messages, events and return statements are from the server; negotiation and execute statements are from the client.) Maybe if we used a hypothetical qmp-shell log format, we could add timestamps here instead of relying on the client to produce them. This might be interesting for analyzing race conditions and measuring response delays as experienced by the server. {"message": original_json_message_here, "direction": "in", "timestamp": 1610627721} (Downside: JSON is still not a streaming message format, but I guess it's one we already use all over the place anyway.)
2. Add a chardev-get-fd command that fetches the fd from a chardev and then use the existing chardev-change command to replace the monitor chardev with a chardev connected to qmp-shell. This inserts qmp-shell as a proxy between the QMP client and server. qmp-shell can remove itself again with another chardev-change command. This approach allows MITM. The downside is it assumes the QMP chardev is a file descriptor, so it won't work with all types of chardev.
It seems a little more prone to failure if the insertion/removal fails, and has some downsides about which configurations it can inject into.
3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin source chardev, 2. a monitoring sink chardev, and 3. a monitoring source chardev. The data flow is origin <-> monitoring sink <-> monitoring source <-> QMP monitor. qmp-shell creates the monitoring sink (for receiving incoming QMP commands) and monitoring source chardev (for forwarding QMP commands or MITM commands), and then it uses change-chardev to instantiate a chardev-proxy that directs the original libvirt chardev through the monitoring sink and source.
I'm not sure I understand the topology here, exactly. I could stand to be a little more familiar with how chardevs are modeled in QEMU ...
This is the most complex but also completely contained within the QEMU chardev layer.
In all these approaches qmp-shell uses virsh qemu-monitor-command or an equivalent API to start/stop monitoring a running VM without manual setup steps.
Gotcha. I think I am leaning towards the first suggestion, but maybe the third one that I don't quite grasp yet is good too.
Stefan
participants (4)
-
Daniel P. Berrangé -
John Snow -
Kevin Wolf -
Stefan Hajnoczi