Qemu monitor info tlb gives unable to encode message payload
Hi, the command virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info tlb" fails with "error: Unable to encode message payload". I found a bugtracker entry for a similiar error [1], but I don't if this is the same error (message too large). I also don't know how large an info tlb message is. Preferably I would not have to recompile libvirt just to issue this monitor command.. Then I thought about circumventing the error by connecting directly to the qemu monitor via netcat, but I found a thread [2] that says I cannot add my own "-monitor tcp:..." to the Qemu commandline arguments. Could someone help me with this? Best, Philipp [1] https://bugzilla.redhat.com/show_bug.cgi?id=1440683 [2] https://libvirt-users.redhat.narkive.com/E0UTmdhU/is-there-a-way-to-disable-...
On Wed, Dec 08, 2021 at 10:30:27 +0000, Philipp Klocke wrote:
Hi,
the command virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info tlb" fails with "error: Unable to encode message payload".
I found a bugtracker entry for a similiar error [1], but I don't if this is the same error (message too large). I also don't know how large an info tlb message is. Preferably I would not have to recompile libvirt just to issue this monitor command..
Libvirt unfortunately limits strings to 4MiB: const REMOTE_STRING_MAX = 4194304; And the reply from qemu-monitor-command is a single string. Now internally we process JSON messages up to 10 MiB so one could argue that we should increase the size for the 'qemu-monitor-command' reply up to 10MiB. This could be straightforward but it's questionable whether it's worth it.
Then I thought about circumventing the error by connecting directly to the qemu monitor via netcat, but I found a thread [2] that says I cannot add my own "-monitor tcp:..." to the Qemu commandline arguments.
IIRC at that point qemu wasn't able to handle two monitor connections. At this point it is possible to have two concurrent connections to the montitor. Obviously things may break and you get to keep the pieces if it breaks. By adding: <qemu:commandline> <qemu:arg value='-qmp'/> <qemu:arg value='tcp:127.0.0.1:1235'/> </qemu:commandline> I then get a connection from qemu on the socket when starting the VM: $ nc -l -p 1235 {"QMP": {"version": {"qemu": {"micro": 93, "minor": 1, "major": 6}, "package": "v6.2.0-rc3"}, "capabilities": ["oob"]}} I can then start conversing with the monitor: {"execute":"qmp_capabilities"} {"return": {}} {"execute":"human-monitor-command","arguments":{"command-line":"info"}} {"return": "info balloon - .... Out of curiousity, what do you specifically need 'info tlb' for?
On 12/8/21 12:09, Peter Krempa wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
On Wed, Dec 08, 2021 at 10:30:27 +0000, Philipp Klocke wrote:
Hi,
the command virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info tlb" fails with "error: Unable to encode message payload".
I found a bugtracker entry for a similiar error [1], but I don't if this is the same error (message too large). I also don't know how large an info tlb message is. Preferably I would not have to recompile libvirt just to issue this monitor command.. Libvirt unfortunately limits strings to 4MiB:
const REMOTE_STRING_MAX = 4194304;
And the reply from qemu-monitor-command is a single string. Now internally we process JSON messages up to 10 MiB so one could argue that we should increase the size for the 'qemu-monitor-command' reply up to 10MiB. This could be straightforward but it's questionable whether it's worth it. Thanks for the clarification! I will try to go for the 2nd monitor then.
Then I thought about circumventing the error by connecting directly to the qemu monitor via netcat, but I found a thread [2] that says I cannot add my own "-monitor tcp:..." to the Qemu commandline arguments. IIRC at that point qemu wasn't able to handle two monitor connections. At this point it is possible to have two concurrent connections to the montitor. Obviously things may break and you get to keep the pieces if it breaks.
By adding:
<qemu:commandline> <qemu:arg value='-qmp'/> <qemu:arg value='tcp:127.0.0.1:1235'/> </qemu:commandline>
When I add this to the config via virsh edit and then do a shutdown + reboot, I get a kernel panic. I put the corresponding dmesg log on gist: https://gist.github.com/klockeph/323f3e0d23a3254ef98a65fd7c8f5a1c
I then get a connection from qemu on the socket when starting the VM:
$ nc -l -p 1235 {"QMP": {"version": {"qemu": {"micro": 93, "minor": 1, "major": 6}, "package": "v6.2.0-rc3"}, "capabilities": ["oob"]}}
I can then start conversing with the monitor:
{"execute":"qmp_capabilities"} {"return": {}} {"execute":"human-monitor-command","arguments":{"command-line":"info"}} {"return": "info balloon - ....
Out of curiousity, what do you specifically need 'info tlb' for?
I am currently investigating a performance issue where a Windows VM is very slow in accessing qemu ivshmem. So I want to get some better understanding how this memory is mapped into the VM (if it's paginated or continuous etc.), and hoped that info tlb could help me there. But I don't know what the output of the command is because it never ran. :D I would prefer using a "real" profiler for this but the only one I tried so far (AMD uProf) crashed the VM with a bluescreen...
On Wed, Dec 08, 2021 at 01:32:48PM +0100, Philipp Klocke wrote:
On 12/8/21 12:09, Peter Krempa wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
On Wed, Dec 08, 2021 at 10:30:27 +0000, Philipp Klocke wrote:
Hi,
the command virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info tlb" fails with "error: Unable to encode message payload".
I found a bugtracker entry for a similiar error [1], but I don't if this is the same error (message too large). I also don't know how large an info tlb message is. Preferably I would not have to recompile libvirt just to issue this monitor command.. Libvirt unfortunately limits strings to 4MiB:
const REMOTE_STRING_MAX = 4194304;
And the reply from qemu-monitor-command is a single string. Now internally we process JSON messages up to 10 MiB so one could argue that we should increase the size for the 'qemu-monitor-command' reply up to 10MiB. This could be straightforward but it's questionable whether it's worth it. Thanks for the clarification! I will try to go for the 2nd monitor then.
Then I thought about circumventing the error by connecting directly to the qemu monitor via netcat, but I found a thread [2] that says I cannot add my own "-monitor tcp:..." to the Qemu commandline arguments. IIRC at that point qemu wasn't able to handle two monitor connections. At this point it is possible to have two concurrent connections to the montitor. Obviously things may break and you get to keep the pieces if it breaks.
By adding:
<qemu:commandline> <qemu:arg value='-qmp'/> <qemu:arg value='tcp:127.0.0.1:1235'/> </qemu:commandline>
When I add this to the config via virsh edit and then do a shutdown + reboot, I get a kernel panic. I put the corresponding dmesg log on gist: https://gist.github.com/klockeph/323f3e0d23a3254ef98a65fd7c8f5a1c
I can't see how that is related to adding an extra -qmp parameter, since it doesn't affect the guest ABI FWIW, exposing -qmp over a tcp socket is totally insecure - any local process can connect to that. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 12/8/21 14:15, Daniel P. Berrangé wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
On Wed, Dec 08, 2021 at 01:32:48PM +0100, Philipp Klocke wrote:
On 12/8/21 12:09, Peter Krempa wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
On Wed, Dec 08, 2021 at 10:30:27 +0000, Philipp Klocke wrote:
Hi,
the command virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info tlb" fails with "error: Unable to encode message payload".
I found a bugtracker entry for a similiar error [1], but I don't if this is the same error (message too large). I also don't know how large an info tlb message is. Preferably I would not have to recompile libvirt just to issue this monitor command.. Libvirt unfortunately limits strings to 4MiB:
const REMOTE_STRING_MAX = 4194304;
And the reply from qemu-monitor-command is a single string. Now internally we process JSON messages up to 10 MiB so one could argue that we should increase the size for the 'qemu-monitor-command' reply up to 10MiB. This could be straightforward but it's questionable whether it's worth it. Thanks for the clarification! I will try to go for the 2nd monitor then.
Then I thought about circumventing the error by connecting directly to the qemu monitor via netcat, but I found a thread [2] that says I cannot add my own "-monitor tcp:..." to the Qemu commandline arguments. IIRC at that point qemu wasn't able to handle two monitor connections. At this point it is possible to have two concurrent connections to the montitor. Obviously things may break and you get to keep the pieces if it breaks.
By adding:
<qemu:commandline> <qemu:arg value='-qmp'/> <qemu:arg value='tcp:127.0.0.1:1235'/> </qemu:commandline>
When I add this to the config via virsh edit and then do a shutdown + reboot, I get a kernel panic. I put the corresponding dmesg log on gist: https://gist.github.com/klockeph/323f3e0d23a3254ef98a65fd7c8f5a1c
I can't see how that is related to adding an extra -qmp parameter, since it doesn't affect the guest ABI
Thanks, it seems that the panic was unrelated and something else that I need to fix.
FWIW, exposing -qmp over a tcp socket is totally insecure - any local process can connect to that.
For some reason the tcp connection didn't work, but unix socket does the trick. I can now issue "info tlb" JSON commands through this socket and get a reply. <qemu:commandline> <qemu:arg value='-qmp'/> <qemu:arg value='unix:/path-to/ubuntu_vm.sock,server,nowait'/> </qemu:commandline> Thanks everybody for the help! Best, Philipp
On 12/8/21 12:09, Peter Krempa wrote
On Wed, Dec 08, 2021 at 10:30:27 +0000, Philipp Klocke wrote:
Hi,
the command virsh qemu-monitor-command ubuntu_vm --hmp --cmd "info tlb" fails with "error: Unable to encode message payload".
I found a bugtracker entry for a similiar error [1], but I don't if this is the same error (message too large). I also don't know how large an info tlb message is. Preferably I would not have to recompile libvirt just to issue this monitor command..
Libvirt unfortunately limits strings to 4MiB:
const REMOTE_STRING_MAX = 4194304;
And the reply from qemu-monitor-command is a single string. Now internally we process JSON messages up to 10 MiB so one could argue that we should increase the size for the 'qemu-monitor-command' reply up to 10MiB. This could be straightforward but it's questionable whether it's worth it.
Thanks for the clarification! I will try to go for the 2nd monitor then.
Then I thought about circumventing the error by connecting directly to the qemu monitor via netcat, but I found a thread [2] that says I cannot add my own "-monitor tcp:..." to the Qemu commandline arguments.
IIRC at that point qemu wasn't able to handle two monitor connections. At this point it is possible to have two concurrent connections to the montitor. Obviously things may break and you get to keep the pieces if it breaks.
By adding:
<qemu:commandline> <qemu:arg value='-qmp'/> <qemu:arg value='tcp:127.0.0.1:1235'/> </qemu:commandline>
When I add this to the config via virsh edit and then do a shutdown + reboot, I get a kernel panic. I put the corresponding dmesg log on gist: https://gist.github.com/klockeph/323f3e0d23a3254ef98a65fd7c8f5a1c
Out of curiousity, what do you specifically need 'info tlb' for?
I am currently investigating a performance issue where a Windows VM is very slow in accessing qemu ivshmem. So I want to get some better understanding how this memory is mapped into the VM (if it's paginated or continuous etc.), and hoped that info tlb could help me there. But I don't know what the output of the command is because it never ran. :D I would prefer using a "real" profiler for this but the only one I tried so far (AMD uProf) crashed the VM with a bluescreen...
participants (3)
-
Daniel P. Berrangé -
Peter Krempa -
Philipp Klocke