[libvirt] error: server response too large
Hi. When trying to do a screenshot of a remote domain connected via qemu+tcp (for testing purposes only), I receive this error: ---------------------------------------------------------------------- virsh -c qemu+tcp://dev/system Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # version Compiled against library: libvir 0.9.8 Using library: libvir 0.9.8 Using API: QEMU 0.9.8 Running hypervisor: QEMU 1.5.1 virsh # screenshot 2 /tmp/screendump error: could not receive data from domain 2 error: packet 1048600 bytes received from server too large, want 262144 virsh # 2013-09-30 14:47:05.158+0000: 21646: info : libvirt version: 0.9.8 2013-09-30 14:47:05.158+0000: 21646: warning : virNetClientIncomingEvent:1660 : Something went wrong during async message processing ---------------------------------------------------------------------- I'm using Ubuntu LTS 12.04.3, latest version in the repo which is 0.9.8-2ubuntu17.13. On the server side, I'm using a self-compiled libvirt from git master. On the server, I receive: 2013-09-30 14:41:10.075+0000: 1203: info : libvirt version: 1.1.2 2013-09-30 14:41:10.075+0000: 1203: error : virNetSocketWriteWire:1446 : Cannot write data: Connection reset by peer 2013-09-30 14:41:10.077+0000: 1203: error : virFDStreamCloseInt:315 : internal error: I/O helper exited abnormally 2013-09-30 14:41:10.078+0000: 1203: error : virFDStreamUpdateCallback:133 : internal error: stream is not open I've uploaded the logs produced with LIBVIRT_DEBUG=1 to sendspace: http://www.sendspace.com/file/tevueo (libvirtd.log.gz) [1MB] http://www.sendspace.com/file/rdpmrh (virsh.log.gz) [5KB] When using a non-async virStream with client library version 1.1.2, it works OK. Is this a bug in libvirt 0.9.8 or is this a regression in >= 1.1.2? I started to git bisect but ran out of time, I'm out of office till next week. Any help would be great. Thanks. Claudio -- AV-Test GmbH, Henricistraße 20, 04155 Leipzig, Germany Phone: +49 341 265 310 19 Web:<http://www.av-test.org> Eingetragen am / Registered at: Amtsgericht Stendal (HRB 114076) Geschaeftsfuehrer (CEO): Andreas Marx, Guido Habicht, Maik Morgenstern
On 30.09.2013 17:02, Claudio Bley wrote:
Hi.
When trying to do a screenshot of a remote domain connected via qemu+tcp (for testing purposes only), I receive this error:
---------------------------------------------------------------------- virsh -c qemu+tcp://dev/system Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands 'quit' to quit
virsh # version Compiled against library: libvir 0.9.8 Using library: libvir 0.9.8 Using API: QEMU 0.9.8 Running hypervisor: QEMU 1.5.1
virsh # screenshot 2 /tmp/screendump error: could not receive data from domain 2 error: packet 1048600 bytes received from server too large, want 262144
virsh # 2013-09-30 14:47:05.158+0000: 21646: info : libvirt version: 0.9.8 2013-09-30 14:47:05.158+0000: 21646: warning : virNetClientIncomingEvent:1660 : Something went wrong during async message processing ----------------------------------------------------------------------
I'm using Ubuntu LTS 12.04.3, latest version in the repo which is 0.9.8-2ubuntu17.13.
This works as expected. 0.9.8 still had a small buffer for RPC messages. It's since 0.9.13 release that we've switched to dynamically allocated buffer and hence could size up the limit for incoming data. Update the client and problem will just go away. Michal
On Mon, Sep 30, 2013 at 05:23:33PM +0200, Michal Privoznik wrote:
On 30.09.2013 17:02, Claudio Bley wrote:
Hi.
When trying to do a screenshot of a remote domain connected via qemu+tcp (for testing purposes only), I receive this error:
---------------------------------------------------------------------- virsh -c qemu+tcp://dev/system Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands 'quit' to quit
virsh # version Compiled against library: libvir 0.9.8 Using library: libvir 0.9.8 Using API: QEMU 0.9.8 Running hypervisor: QEMU 1.5.1
virsh # screenshot 2 /tmp/screendump error: could not receive data from domain 2 error: packet 1048600 bytes received from server too large, want 262144
virsh # 2013-09-30 14:47:05.158+0000: 21646: info : libvirt version: 0.9.8 2013-09-30 14:47:05.158+0000: 21646: warning : virNetClientIncomingEvent:1660 : Something went wrong during async message processing ----------------------------------------------------------------------
I'm using Ubuntu LTS 12.04.3, latest version in the repo which is 0.9.8-2ubuntu17.13.
This works as expected. 0.9.8 still had a small buffer for RPC messages. It's since 0.9.13 release that we've switched to dynamically allocated buffer and hence could size up the limit for incoming data. Update the client and problem will just go away.
Actually that is not working as expected. The old client should *always* be able to talk to a new server. If the new server is unconditionally sending back data that is too large for the client, this is a bug. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 09/30/2013 05:24 PM, Daniel P. Berrange wrote:
0.9.8-2ubuntu17.13.
This works as expected. 0.9.8 still had a small buffer for RPC messages. It's since 0.9.13 release that we've switched to dynamically allocated buffer and hence could size up the limit for incoming data. Update the client and problem will just go away.
Actually that is not working as expected. The old client should *always* be able to talk to a new server. If the new server is unconditionally sending back data that is too large for the client, this is a bug.
True, however I don't see how to fix it for past versions, specifically we can't go back to the small buffers again. For a proper solution client and server would need to advertise their respective maximum buffer sizes (or know it by the version number) and restrict themselves to using the lower number. -- Mit freundlichen Grüßen/Kind Regards Viktor Mihajlovski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Köderitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On Mon, Sep 30, 2013 at 04:24:49PM +0100, Daniel P. Berrange wrote:
On Mon, Sep 30, 2013 at 05:23:33PM +0200, Michal Privoznik wrote:
On 30.09.2013 17:02, Claudio Bley wrote:
Hi.
When trying to do a screenshot of a remote domain connected via qemu+tcp (for testing purposes only), I receive this error:
---------------------------------------------------------------------- virsh -c qemu+tcp://dev/system Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands 'quit' to quit
virsh # version Compiled against library: libvir 0.9.8 Using library: libvir 0.9.8 Using API: QEMU 0.9.8 Running hypervisor: QEMU 1.5.1
virsh # screenshot 2 /tmp/screendump error: could not receive data from domain 2 error: packet 1048600 bytes received from server too large, want 262144
virsh # 2013-09-30 14:47:05.158+0000: 21646: info : libvirt version: 0.9.8 2013-09-30 14:47:05.158+0000: 21646: warning : virNetClientIncomingEvent:1660 : Something went wrong during async message processing ----------------------------------------------------------------------
I'm using Ubuntu LTS 12.04.3, latest version in the repo which is 0.9.8-2ubuntu17.13.
This works as expected. 0.9.8 still had a small buffer for RPC messages. It's since 0.9.13 release that we've switched to dynamically allocated buffer and hence could size up the limit for incoming data. Update the client and problem will just go away.
Actually that is not working as expected. The old client should *always* be able to talk to a new server. If the new server is unconditionally sending back data that is too large for the client, this is a bug.
This problem was fixed in commit 27e81517a876dcb738dd8a9bb2e0e68d71c3b7e3 Author: Daniel P. Berrange <berrange@redhat.com> Date: Mon Sep 30 17:27:51 2013 +0100 Fix max stream packet size for old clients The libvirtd server pushes data out to clients. It does not know what protocol version the client might have, so must be conservative and use the old payload limits. ie send no more than 256kb of data per packet. And backported to stable branches. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 10/01/2013 03:02 PM, Daniel P. Berrange wrote:
On Mon, Sep 30, 2013 at 04:24:49PM +0100, Daniel P. Berrange wrote:
On Mon, Sep 30, 2013 at 05:23:33PM +0200, Michal Privoznik wrote:
On 30.09.2013 17:02, Claudio Bley wrote:
Hi.
When trying to do a screenshot of a remote domain connected via qemu+tcp (for testing purposes only), I receive this error:
---------------------------------------------------------------------- virsh -c qemu+tcp://dev/system Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands 'quit' to quit
virsh # version Compiled against library: libvir 0.9.8 Using library: libvir 0.9.8 Using API: QEMU 0.9.8 Running hypervisor: QEMU 1.5.1
virsh # screenshot 2 /tmp/screendump error: could not receive data from domain 2 error: packet 1048600 bytes received from server too large, want 262144
virsh # 2013-09-30 14:47:05.158+0000: 21646: info : libvirt version: 0.9.8 2013-09-30 14:47:05.158+0000: 21646: warning : virNetClientIncomingEvent:1660 : Something went wrong during async message processing ----------------------------------------------------------------------
I'm using Ubuntu LTS 12.04.3, latest version in the repo which is 0.9.8-2ubuntu17.13.
This works as expected. 0.9.8 still had a small buffer for RPC messages. It's since 0.9.13 release that we've switched to dynamically allocated buffer and hence could size up the limit for incoming data. Update the client and problem will just go away.
Actually that is not working as expected. The old client should *always* be able to talk to a new server. If the new server is unconditionally sending back data that is too large for the client, this is a bug.
This problem was fixed in
commit 27e81517a876dcb738dd8a9bb2e0e68d71c3b7e3 Author: Daniel P. Berrange <berrange@redhat.com> Date: Mon Sep 30 17:27:51 2013 +0100
Fix max stream packet size for old clients
The libvirtd server pushes data out to clients. It does not know what protocol version the client might have, so must be conservative and use the old payload limits. ie send no more than 256kb of data per packet.
And backported to stable branches.
Daniel
I tried libvirt 1.1.3 containing the commit on the server and Ubuntu libvirt 0.9.8 as the client and still have the issue if the message is greater than 256KB. The problem is that the old client checks the message size during decode (which is greater than the client's max message size). Not sure how to get out of this ... -- Mit freundlichen Grüßen/Kind Regards Viktor Mihajlovski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Köderitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On Tue, Oct 01, 2013 at 06:13:10PM +0200, Viktor Mihajlovski wrote:
On 10/01/2013 03:02 PM, Daniel P. Berrange wrote:
On Mon, Sep 30, 2013 at 04:24:49PM +0100, Daniel P. Berrange wrote:
On Mon, Sep 30, 2013 at 05:23:33PM +0200, Michal Privoznik wrote:
On 30.09.2013 17:02, Claudio Bley wrote:
Hi.
When trying to do a screenshot of a remote domain connected via qemu+tcp (for testing purposes only), I receive this error:
---------------------------------------------------------------------- virsh -c qemu+tcp://dev/system Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands 'quit' to quit
virsh # version Compiled against library: libvir 0.9.8 Using library: libvir 0.9.8 Using API: QEMU 0.9.8 Running hypervisor: QEMU 1.5.1
virsh # screenshot 2 /tmp/screendump error: could not receive data from domain 2 error: packet 1048600 bytes received from server too large, want 262144
virsh # 2013-09-30 14:47:05.158+0000: 21646: info : libvirt version: 0.9.8 2013-09-30 14:47:05.158+0000: 21646: warning : virNetClientIncomingEvent:1660 : Something went wrong during async message processing ----------------------------------------------------------------------
I'm using Ubuntu LTS 12.04.3, latest version in the repo which is 0.9.8-2ubuntu17.13.
This works as expected. 0.9.8 still had a small buffer for RPC messages. It's since 0.9.13 release that we've switched to dynamically allocated buffer and hence could size up the limit for incoming data. Update the client and problem will just go away.
Actually that is not working as expected. The old client should *always* be able to talk to a new server. If the new server is unconditionally sending back data that is too large for the client, this is a bug.
This problem was fixed in
commit 27e81517a876dcb738dd8a9bb2e0e68d71c3b7e3 Author: Daniel P. Berrange <berrange@redhat.com> Date: Mon Sep 30 17:27:51 2013 +0100
Fix max stream packet size for old clients
The libvirtd server pushes data out to clients. It does not know what protocol version the client might have, so must be conservative and use the old payload limits. ie send no more than 256kb of data per packet.
And backported to stable branches.
Daniel
I tried libvirt 1.1.3 containing the commit on the server and Ubuntu libvirt 0.9.8 as the client and still have the issue if the message is greater than 256KB. The problem is that the old client checks the message size during decode (which is greater than the client's max message size).
Not sure how to get out of this ...
What were you doing to get a message greater than 256KB ? This patch did not attempt to fix all possible combinations. If a API call such as virConnectListAllDomains has so much data that it requires > 256KB to encode, this fix won't solve that. There is nothing we can do for API calls which have a genuine need to exceed the old size limit. I was only addressing the case of virStreamPtr data which was mistakenly hardcoded in the server to try sending 16 MB of data at once. I switched it to only try to send 256 KB of data per stream packet. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 10/01/2013 06:19 PM, Daniel P. Berrange wrote:
What were you doing to get a message greater than 256KB ? This patch did not attempt to fix all possible combinations. If a API call such as virConnectListAllDomains has so much data that it requires > 256KB to encode, this fix won't solve that. There is nothing we can do for API calls which have a genuine need to exceed the old size limit.
I was only addressing the case of virStreamPtr data which was mistakenly hardcoded in the server to try sending 16 MB of data at once. I switched it to only try to send 256 KB of data per stream packet.
OK, than it's a misunderstanding from my part (regarding your intention). The client side error message was the same as in the "all-at-once" case which would mean that the client somehow got past receiving the message even then and finally stumbled trying to decode the message (which is inevitable in such a case). Our S390 domains happen to be "oversize" due to the large number of devices (which is harder to reproduce on a PCI based system because it runs out of bus addresses too quick). So dumpxml typically raises the condition. With SCSI it's easier to create large guests, so a script like the one below can help to reproduce... #!/bin/bash cat > scsidom <<EOF <domain type='kvm'> <name>manydevs</name> <memory>1048576</memory> <os> <type>hvm</type> </os> <devices> <controller type='scsi' model='virtio-scsi'/> EOF for i in {1..1000} do cat >> scsidom <<EOF <disk type='file' device='disk'> <source file='/var/lib/libvirt/images/vol$i'/> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='$i'/> </disk> EOF done cat >> scsidom <<EOF </devices> </domain> EOF -- Mit freundlichen Grüßen/Kind Regards Viktor Mihajlovski IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Köderitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
On Tue, Oct 01, 2013 at 06:57:56PM +0200, Viktor Mihajlovski wrote:
On 10/01/2013 06:19 PM, Daniel P. Berrange wrote:
What were you doing to get a message greater than 256KB ? This patch did not attempt to fix all possible combinations. If a API call such as virConnectListAllDomains has so much data that it requires > 256KB to encode, this fix won't solve that. There is nothing we can do for API calls which have a genuine need to exceed the old size limit.
I was only addressing the case of virStreamPtr data which was mistakenly hardcoded in the server to try sending 16 MB of data at once. I switched it to only try to send 256 KB of data per stream packet.
OK, than it's a misunderstanding from my part (regarding your intention). The client side error message was the same as in the "all-at-once" case which would mean that the client somehow got past receiving the message even then and finally stumbled trying to decode the message (which is inevitable in such a case).
Our S390 domains happen to be "oversize" due to the large number of devices (which is harder to reproduce on a PCI based system because it runs out of bus addresses too quick). So dumpxml typically raises the condition.
Yep, there's nothing we can do about that scenario with old clients, since the data we're returning is inherantly too big. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
At Tue, 1 Oct 2013 17:19:56 +0100, Daniel P. Berrange wrote:
On Tue, Oct 01, 2013 at 06:13:10PM +0200, Viktor Mihajlovski wrote:
On 10/01/2013 03:02 PM, Daniel P. Berrange wrote:
On Mon, Sep 30, 2013 at 04:24:49PM +0100, Daniel P. Berrange wrote:
On Mon, Sep 30, 2013 at 05:23:33PM +0200, Michal Privoznik wrote:
On 30.09.2013 17:02, Claudio Bley wrote:
Hi.
When trying to do a screenshot of a remote domain connected via qemu+tcp (for testing purposes only), I receive this error:
---------------------------------------------------------------------- virsh -c qemu+tcp://dev/system Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands 'quit' to quit
virsh # version Compiled against library: libvir 0.9.8 Using library: libvir 0.9.8 Using API: QEMU 0.9.8 Running hypervisor: QEMU 1.5.1
virsh # screenshot 2 /tmp/screendump error: could not receive data from domain 2 error: packet 1048600 bytes received from server too large, want 262144
virsh # 2013-09-30 14:47:05.158+0000: 21646: info : libvirt version: 0.9.8 2013-09-30 14:47:05.158+0000: 21646: warning : virNetClientIncomingEvent:1660 : Something went wrong during async message processing ----------------------------------------------------------------------
I'm using Ubuntu LTS 12.04.3, latest version in the repo which is 0.9.8-2ubuntu17.13.
This problem was fixed in
commit 27e81517a876dcb738dd8a9bb2e0e68d71c3b7e3 Author: Daniel P. Berrange <berrange@redhat.com> Date: Mon Sep 30 17:27:51 2013 +0100
Fix max stream packet size for old clients
The libvirtd server pushes data out to clients. It does not know what protocol version the client might have, so must be conservative and use the old payload limits. ie send no more than 256kb of data per packet.
And backported to stable branches.
I tried libvirt 1.1.3 containing the commit on the server and Ubuntu libvirt 0.9.8 as the client and still have the issue if the message is greater than 256KB. The problem is that the old client checks the message size during decode (which is greater than the client's max message size).
Not sure how to get out of this ...
What were you doing to get a message greater than 256KB ? This patch did not attempt to fix all possible combinations. If a API call such as virConnectListAllDomains has so much data that it requires > 256KB to encode, this fix won't solve that. There is nothing we can do for API calls which have a genuine need to exceed the old size limit.
I was only addressing the case of virStreamPtr data which was mistakenly hardcoded in the server to try sending 16 MB of data at once. I switched it to only try to send 256 KB of data per stream packet.
I've also tested your patch, and it seems 256 KB of data is still a bit too large: virsh # screenshot 2 /tmp/test error: could not receive data from domain 2 error: packet 262168 bytes received from server too large, want 262144 The max payload size is computed as: VIR_NET_MESSAGE_MAX = 16777216 VIR_NET_MESSAGE_HEADER_MAX = 24 VIR_NET_MESSAGE_PAYLOAD_MAX = (VIR_NET_MESSAGE_MAX - VIR_NET_MESSAGE_HEADER_MAX) = 16777192 So, it seems the legacy max payload size is actually 262120; I tested it and it works. ------------------- 8< ------ >8 --------------------------- Subject: [PATCH] Adjust legacy max payload size to account for header information Organization: AV-Test GmbH, Germany Commit 27e81517a87 set the payload size to 256 KB, which is actually the max packet size, including the size of the header. Reduce this by VIR_NET_MESSAGE_HEADER_MAX (24) and set VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX to 262120. Signed-off-by: Claudio Bley <cbley@av-test.de> --- src/rpc/virnetprotocol.x | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rpc/virnetprotocol.x b/src/rpc/virnetprotocol.x index 1eae7cb..7b6f753 100644 --- a/src/rpc/virnetprotocol.x +++ b/src/rpc/virnetprotocol.x @@ -55,7 +55,7 @@ const VIR_NET_MESSAGE_INITIAL = 65536; * payload size. We need to remember this for compat with * old clients. */ -const VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX = 262144; +const VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX = 262120; /* Maximum total message size (serialised). */ const VIR_NET_MESSAGE_MAX = 16777216; ------------------- 8< ------ >8 --------------------------- -- AV-Test GmbH, Henricistraße 20, 04155 Leipzig, Germany Phone: +49 341 265 310 19 Web:<http://www.av-test.org> Eingetragen am / Registered at: Amtsgericht Stendal (HRB 114076) Geschaeftsfuehrer (CEO): Andreas Marx, Guido Habicht, Maik Morgenstern
On Mon, Oct 07, 2013 at 12:23:44PM +0200, Claudio Bley wrote:
At Tue, 1 Oct 2013 17:19:56 +0100, Daniel P. Berrange wrote:
On Tue, Oct 01, 2013 at 06:13:10PM +0200, Viktor Mihajlovski wrote:
On 10/01/2013 03:02 PM, Daniel P. Berrange wrote:
On Mon, Sep 30, 2013 at 04:24:49PM +0100, Daniel P. Berrange wrote:
On Mon, Sep 30, 2013 at 05:23:33PM +0200, Michal Privoznik wrote:
On 30.09.2013 17:02, Claudio Bley wrote: >Hi. > >When trying to do a screenshot of a remote domain connected via >qemu+tcp (for testing purposes only), I receive this error: > >---------------------------------------------------------------------- >virsh -c qemu+tcp://dev/system >Welcome to virsh, the virtualization interactive terminal. > >Type: 'help' for help with commands > 'quit' to quit > >virsh # version >Compiled against library: libvir 0.9.8 >Using library: libvir 0.9.8 >Using API: QEMU 0.9.8 >Running hypervisor: QEMU 1.5.1 > >virsh # screenshot 2 /tmp/screendump >error: could not receive data from domain 2 >error: packet 1048600 bytes received from server too large, want 262144 > >virsh # 2013-09-30 14:47:05.158+0000: 21646: info : libvirt version: 0.9.8 >2013-09-30 14:47:05.158+0000: 21646: warning : virNetClientIncomingEvent:1660 : Something went wrong during async message processing >---------------------------------------------------------------------- > >I'm using Ubuntu LTS 12.04.3, latest version in the repo which is >0.9.8-2ubuntu17.13.
This problem was fixed in
commit 27e81517a876dcb738dd8a9bb2e0e68d71c3b7e3 Author: Daniel P. Berrange <berrange@redhat.com> Date: Mon Sep 30 17:27:51 2013 +0100
Fix max stream packet size for old clients
The libvirtd server pushes data out to clients. It does not know what protocol version the client might have, so must be conservative and use the old payload limits. ie send no more than 256kb of data per packet.
And backported to stable branches.
I tried libvirt 1.1.3 containing the commit on the server and Ubuntu libvirt 0.9.8 as the client and still have the issue if the message is greater than 256KB. The problem is that the old client checks the message size during decode (which is greater than the client's max message size).
Not sure how to get out of this ...
What were you doing to get a message greater than 256KB ? This patch did not attempt to fix all possible combinations. If a API call such as virConnectListAllDomains has so much data that it requires > 256KB to encode, this fix won't solve that. There is nothing we can do for API calls which have a genuine need to exceed the old size limit.
I was only addressing the case of virStreamPtr data which was mistakenly hardcoded in the server to try sending 16 MB of data at once. I switched it to only try to send 256 KB of data per stream packet.
I've also tested your patch, and it seems 256 KB of data is still a bit too large:
virsh # screenshot 2 /tmp/test error: could not receive data from domain 2 error: packet 262168 bytes received from server too large, want 262144
The max payload size is computed as:
VIR_NET_MESSAGE_MAX = 16777216 VIR_NET_MESSAGE_HEADER_MAX = 24 VIR_NET_MESSAGE_PAYLOAD_MAX = (VIR_NET_MESSAGE_MAX - VIR_NET_MESSAGE_HEADER_MAX) = 16777192
So, it seems the legacy max payload size is actually 262120; I tested it and it works.
------------------- 8< ------ >8 --------------------------- Subject: [PATCH] Adjust legacy max payload size to account for header information Organization: AV-Test GmbH, Germany
Commit 27e81517a87 set the payload size to 256 KB, which is actually the max packet size, including the size of the header.
Reduce this by VIR_NET_MESSAGE_HEADER_MAX (24) and set VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX to 262120.
Signed-off-by: Claudio Bley <cbley@av-test.de> --- src/rpc/virnetprotocol.x | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/rpc/virnetprotocol.x b/src/rpc/virnetprotocol.x index 1eae7cb..7b6f753 100644 --- a/src/rpc/virnetprotocol.x +++ b/src/rpc/virnetprotocol.x @@ -55,7 +55,7 @@ const VIR_NET_MESSAGE_INITIAL = 65536; * payload size. We need to remember this for compat with * old clients. */ -const VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX = 262144; +const VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX = 262120;
/* Maximum total message size (serialised). */ const VIR_NET_MESSAGE_MAX = 16777216;
Damn, yes, you are correct. The original value was 262120, I copied the wrong value. THe commit which changed it first was commit eb635de1fed3257c5c62b552d1ec981c9545c1d7 Author: Michal Privoznik <mprivozn@redhat.com> Date: Fri Apr 27 14:49:48 2012 +0200 rpc: Size up RPC limits /* Size of message payload */ -const VIR_NET_MESSAGE_PAYLOAD_MAX = 262120; +const VIR_NET_MESSAGE_PAYLOAD_MAX = 4194280; Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
At Mon, 7 Oct 2013 11:31:48 +0100, Daniel P. Berrange wrote:
I've also tested your patch, and it seems 256 KB of data is still a bit too large:
virsh # screenshot 2 /tmp/test error: could not receive data from domain 2 error: packet 262168 bytes received from server too large, want 262144
The max payload size is computed as:
VIR_NET_MESSAGE_MAX = 16777216 VIR_NET_MESSAGE_HEADER_MAX = 24 VIR_NET_MESSAGE_PAYLOAD_MAX = (VIR_NET_MESSAGE_MAX - VIR_NET_MESSAGE_HEADER_MAX) = 16777192
So, it seems the legacy max payload size is actually 262120; I tested it and it works.
------------------- 8< ------ >8 --------------------------- Subject: [PATCH] Adjust legacy max payload size to account for header information Organization: AV-Test GmbH, Germany
Commit 27e81517a87 set the payload size to 256 KB, which is actually the max packet size, including the size of the header.
Reduce this by VIR_NET_MESSAGE_HEADER_MAX (24) and set VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX to 262120.
Signed-off-by: Claudio Bley <cbley@av-test.de> --- src/rpc/virnetprotocol.x | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/rpc/virnetprotocol.x b/src/rpc/virnetprotocol.x index 1eae7cb..7b6f753 100644 --- a/src/rpc/virnetprotocol.x +++ b/src/rpc/virnetprotocol.x @@ -55,7 +55,7 @@ const VIR_NET_MESSAGE_INITIAL = 65536; * payload size. We need to remember this for compat with * old clients. */ -const VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX = 262144; +const VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX = 262120;
/* Maximum total message size (serialised). */ const VIR_NET_MESSAGE_MAX = 16777216;
Damn, yes, you are correct. The original value was 262120, I copied the wrong value.
THe commit which changed it first was
commit eb635de1fed3257c5c62b552d1ec981c9545c1d7 Author: Michal Privoznik <mprivozn@redhat.com> Date: Fri Apr 27 14:49:48 2012 +0200
rpc: Size up RPC limits
/* Size of message payload */ -const VIR_NET_MESSAGE_PAYLOAD_MAX = 262120; +const VIR_NET_MESSAGE_PAYLOAD_MAX = 4194280;
I amended my commit message referring to Michal's commit and pushed. Are you pushing to the maintenance branches, then? Claudio -- AV-Test GmbH, Henricistraße 20, 04155 Leipzig, Germany Phone: +49 341 265 310 19 Web:<http://www.av-test.org> Eingetragen am / Registered at: Amtsgericht Stendal (HRB 114076) Geschaeftsfuehrer (CEO): Andreas Marx, Guido Habicht, Maik Morgenstern
On Mon, Oct 07, 2013 at 01:35:56PM +0200, Claudio Bley wrote:
At Mon, 7 Oct 2013 11:31:48 +0100, Daniel P. Berrange wrote:
I've also tested your patch, and it seems 256 KB of data is still a bit too large:
virsh # screenshot 2 /tmp/test error: could not receive data from domain 2 error: packet 262168 bytes received from server too large, want 262144
The max payload size is computed as:
VIR_NET_MESSAGE_MAX = 16777216 VIR_NET_MESSAGE_HEADER_MAX = 24 VIR_NET_MESSAGE_PAYLOAD_MAX = (VIR_NET_MESSAGE_MAX - VIR_NET_MESSAGE_HEADER_MAX) = 16777192
So, it seems the legacy max payload size is actually 262120; I tested it and it works.
------------------- 8< ------ >8 --------------------------- Subject: [PATCH] Adjust legacy max payload size to account for header information Organization: AV-Test GmbH, Germany
Commit 27e81517a87 set the payload size to 256 KB, which is actually the max packet size, including the size of the header.
Reduce this by VIR_NET_MESSAGE_HEADER_MAX (24) and set VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX to 262120.
Signed-off-by: Claudio Bley <cbley@av-test.de> --- src/rpc/virnetprotocol.x | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/rpc/virnetprotocol.x b/src/rpc/virnetprotocol.x index 1eae7cb..7b6f753 100644 --- a/src/rpc/virnetprotocol.x +++ b/src/rpc/virnetprotocol.x @@ -55,7 +55,7 @@ const VIR_NET_MESSAGE_INITIAL = 65536; * payload size. We need to remember this for compat with * old clients. */ -const VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX = 262144; +const VIR_NET_MESSAGE_LEGACY_PAYLOAD_MAX = 262120;
/* Maximum total message size (serialised). */ const VIR_NET_MESSAGE_MAX = 16777216;
Damn, yes, you are correct. The original value was 262120, I copied the wrong value.
THe commit which changed it first was
commit eb635de1fed3257c5c62b552d1ec981c9545c1d7 Author: Michal Privoznik <mprivozn@redhat.com> Date: Fri Apr 27 14:49:48 2012 +0200
rpc: Size up RPC limits
/* Size of message payload */ -const VIR_NET_MESSAGE_PAYLOAD_MAX = 262120; +const VIR_NET_MESSAGE_PAYLOAD_MAX = 4194280;
I amended my commit message referring to Michal's commit and pushed.
Are you pushing to the maintenance branches, then?
Yep, I'll update them. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (4)
-
Claudio Bley -
Daniel P. Berrange -
Michal Privoznik -
Viktor Mihajlovski