[PATCH] In virFDStreamRead(), fill buffer from this and next messages
Before this change, buffers returned from virFDStreamRead() would alternate in size (262120 and 24), because it only consumed the bytes remaining from the current background thread message. As the background thread reads 262144 bytes (256kB) of data in each chunk, where the maximum size returned from virFDStreamRead() to be transferred over the remote protocol is only 262120, 24 bytes would be left in the buffer on each iteration. The next iteration leaves 24 bytes, which used to be returned without considering messages waiting in the queue. Signed-off-by: Erik Huelsmann <ehuels@gmail.com> --- src/util/virfdstream.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/src/util/virfdstream.c b/src/util/virfdstream.c index 26a1f00..4c974ba 100644 --- a/src/util/virfdstream.c +++ b/src/util/virfdstream.c @@ -905,7 +905,10 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) if (fdst->thread) { virFDStreamMsg *msg = NULL; + size_t got = 0; + size_t bsz = 0; + more: while (!(msg = fdst->msg)) { if (fdst->threadQuit || fdst->threadErr) { if (nbytes) { @@ -917,7 +920,7 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) virReportSystemError(EBADF, "%s", _("stream is not open")); } else { - ret = 0; + ret = got; } goto cleanup; } else { @@ -931,7 +934,7 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) * return 0 immediately. */ if (msg->type == VIR_FDSTREAM_MSG_TYPE_HOLE && msg->stream.hole.len == 0) { - ret = 0; + ret = got; goto cleanup; } @@ -942,21 +945,26 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) goto cleanup; } - if (nbytes > msg->stream.data.len - msg->stream.data.offset) - nbytes = msg->stream.data.len - msg->stream.data.offset; + bsz = msg->stream.data.len - msg->stream.data.offset; + if (nbytes < bsz) + bsz = nbytes; - memcpy(bytes, + memcpy(bytes + got, msg->stream.data.buf + msg->stream.data.offset, - nbytes); + bsz); + got += bsz; + nbytes -= bsz; - msg->stream.data.offset += nbytes; + msg->stream.data.offset += bsz; if (msg->stream.data.offset == msg->stream.data.len) { virFDStreamMsgQueuePop(fdst, fdst->fd, "pipe"); virFDStreamMsgFree(msg); } - ret = nbytes; - + ret = got; + if (nbytes > 0) { + goto more; + } } else { retry: ret = read(fdst->fd, bytes, nbytes); -- 2.43.0
When I look in the archives, it seems like the patch got wrapped. Not sure if that's actually the case, because in my sent items folder, the lines in the patch are not wrapped. Since I wasn't set up for a patch-on-mailing list workflow, it'll take a bit of getting used to, if this submission wasn't up to standards. Regards, Erik. On 2/3/26 23:22, Erik Hulsmann wrote:
Before this change, buffers returned from virFDStreamRead() would alternate in size (262120 and 24), because it only consumed the bytes remaining from the current background thread message.
As the background thread reads 262144 bytes (256kB) of data in each chunk, where the maximum size returned from virFDStreamRead() to be transferred over the remote protocol is only 262120, 24 bytes would be left in the buffer on each iteration. The next iteration leaves 24 bytes, which used to be returned without considering messages waiting in the queue.
Signed-off-by: Erik Huelsmann <ehuels@gmail.com> --- src/util/virfdstream.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-)
diff --git a/src/util/virfdstream.c b/src/util/virfdstream.c index 26a1f00..4c974ba 100644 --- a/src/util/virfdstream.c +++ b/src/util/virfdstream.c @@ -905,7 +905,10 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) if (fdst->thread) { virFDStreamMsg *msg = NULL; + size_t got = 0; + size_t bsz = 0; + more: while (!(msg = fdst->msg)) { if (fdst->threadQuit || fdst->threadErr) { if (nbytes) { @@ -917,7 +920,7 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) virReportSystemError(EBADF, "%s", _("stream is not open")); } else { - ret = 0; + ret = got; } goto cleanup; } else { @@ -931,7 +934,7 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) * return 0 immediately. */ if (msg->type == VIR_FDSTREAM_MSG_TYPE_HOLE && msg->stream.hole.len == 0) { - ret = 0; + ret = got; goto cleanup; } @@ -942,21 +945,26 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) goto cleanup; } - if (nbytes > msg->stream.data.len - msg->stream.data.offset) - nbytes = msg->stream.data.len - msg->stream.data.offset; + bsz = msg->stream.data.len - msg->stream.data.offset; + if (nbytes < bsz) + bsz = nbytes; - memcpy(bytes, + memcpy(bytes + got, msg->stream.data.buf + msg->stream.data.offset, - nbytes); + bsz); + got += bsz; + nbytes -= bsz; - msg->stream.data.offset += nbytes; + msg->stream.data.offset += bsz; if (msg->stream.data.offset == msg->stream.data.len) { virFDStreamMsgQueuePop(fdst, fdst->fd, "pipe"); virFDStreamMsgFree(msg); } - ret = nbytes; - + ret = got; + if (nbytes > 0) { + goto more; + } } else { retry: ret = read(fdst->fd, bytes, nbytes);
On Wed, Feb 04, 2026 at 09:32:48AM +0100, Erik Hulsmann wrote:
When I look in the archives, it seems like the patch got wrapped. Not sure if that's actually the case, because in my sent items folder, the lines in the patch are not wrapped. Since I wasn't set up for a patch-on-mailing list workflow, it'll take a bit of getting used to, if this submission wasn't up to standards.
Yes, the patch looks a bit mangled on the list too I'm afraid. Generally I'd suggest using 'git-publish' to send patches to the list, as that'll bypass your mail client entirely. Just configure it to point to your outbound SMTP server. git config --global sendemail.smtpServer stmp.youremailprovider.net and then from the branch with your changes, just run 'git-publish' with no args, and it'll guide you to send the patches on that branch to the list.
Regards,
Erik.
On 2/3/26 23:22, Erik Hulsmann wrote:
Before this change, buffers returned from virFDStreamRead() would alternate in size (262120 and 24), because it only consumed the bytes remaining from the current background thread message.
As the background thread reads 262144 bytes (256kB) of data in each chunk, where the maximum size returned from virFDStreamRead() to be transferred over the remote protocol is only 262120, 24 bytes would be left in the buffer on each iteration. The next iteration leaves 24 bytes, which used to be returned without considering messages waiting in the queue.
Signed-off-by: Erik Huelsmann <ehuels@gmail.com> --- src/util/virfdstream.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-)
diff --git a/src/util/virfdstream.c b/src/util/virfdstream.c index 26a1f00..4c974ba 100644 --- a/src/util/virfdstream.c +++ b/src/util/virfdstream.c @@ -905,7 +905,10 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) if (fdst->thread) { virFDStreamMsg *msg = NULL; + size_t got = 0; + size_t bsz = 0; + more: while (!(msg = fdst->msg)) { if (fdst->threadQuit || fdst->threadErr) { if (nbytes) { @@ -917,7 +920,7 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) virReportSystemError(EBADF, "%s", _("stream is not open")); } else { - ret = 0; + ret = got; } goto cleanup; } else { @@ -931,7 +934,7 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) * return 0 immediately. */ if (msg->type == VIR_FDSTREAM_MSG_TYPE_HOLE && msg->stream.hole.len == 0) { - ret = 0; + ret = got; goto cleanup; } @@ -942,21 +945,26 @@ static int virFDStreamRead(virStreamPtr st, char *bytes, size_t nbytes) goto cleanup; } - if (nbytes > msg->stream.data.len - msg->stream.data.offset) - nbytes = msg->stream.data.len - msg->stream.data.offset; + bsz = msg->stream.data.len - msg->stream.data.offset; + if (nbytes < bsz) + bsz = nbytes; - memcpy(bytes, + memcpy(bytes + got, msg->stream.data.buf + msg->stream.data.offset, - nbytes); + bsz); + got += bsz; + nbytes -= bsz; - msg->stream.data.offset += nbytes; + msg->stream.data.offset += bsz; if (msg->stream.data.offset == msg->stream.data.len) { virFDStreamMsgQueuePop(fdst, fdst->fd, "pipe"); virFDStreamMsgFree(msg); } - ret = nbytes; - + ret = got; + if (nbytes > 0) { + goto more; + } } else { retry: ret = read(fdst->fd, bytes, nbytes);
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 2/11/26 12:46, Daniel P. Berrangé wrote:
On Wed, Feb 04, 2026 at 09:32:48AM +0100, Erik Hulsmann wrote:
When I look in the archives, it seems like the patch got wrapped. Not sure if that's actually the case, because in my sent items folder, the lines in the patch are not wrapped. Since I wasn't set up for a patch-on-mailing list workflow, it'll take a bit of getting used to, if this submission wasn't up to standards. Yes, the patch looks a bit mangled on the list too I'm afraid.
Generally I'd suggest using 'git-publish' to send patches to the list, as that'll bypass your mail client entirely. Just configure it to point to your outbound SMTP server.
Thanks for these pointers! I'll resend. Regards, Erik.
participants (2)
-
Daniel P. Berrangé -
Erik Hulsmann