3:57 a.m.
Hi,
-----Original Message-----
From: Peter Xu [mailto:peterx@redhat.com]
Sent: Thursday, June 6, 2024 5:19 AM
To: Dr. David Alan Gilbert <dave(a)treblig.org>
Cc: Michael Galaxy <mgalaxy(a)akamai.com>; zhengchuan
<zhengchuan(a)huawei.com>; Gonglei (Arei) <arei.gonglei(a)huawei.com>;
Daniel P. Berrangé <berrange(a)redhat.com>; Markus Armbruster
<armbru(a)redhat.com>; Yu Zhang <yu.zhang(a)ionos.com>; Zhijian Li (Fujitsu)
<lizhijian(a)fujitsu.com>; Jinpu Wang <jinpu.wang(a)ionos.com>; Elmar Gerdes
<elmar.gerdes(a)ionos.com>; qemu-devel(a)nongnu.org; Yuval Shaia
<yuval.shaia.ml(a)gmail.com>; Kevin Wolf <kwolf(a)redhat.com>; Prasanna
Kumar Kalever <prasanna.kalever(a)redhat.com>; Cornelia Huck
<cohuck(a)redhat.com>; Michael Roth <michael.roth(a)amd.com>; Prasanna
Kumar Kalever <prasanna4324(a)gmail.com>; integration(a)gluster.org; Paolo
Bonzini <pbonzini(a)redhat.com>; qemu-block(a)nongnu.org;
devel(a)lists.libvirt.org; Hanna Reitz <hreitz(a)redhat.com>; Michael S. Tsirkin
<mst(a)redhat.com>; Thomas Huth <thuth(a)redhat.com>; Eric Blake
<eblake(a)redhat.com>; Song Gao <gaosong(a)loongson.cn>; Marc-André
Lureau <marcandre.lureau(a)redhat.com>; Alex Bennée
<alex.bennee(a)linaro.org>; Wainer dos Santos Moschetta
<wainersm(a)redhat.com>; Beraldo Leal <bleal(a)redhat.com>; Pannengyuan
<pannengyuan(a)huawei.com>; Xiexiangyou <xiexiangyou(a)huawei.com>
Subject: Re: [PATCH-for-9.1 v2 2/3] migration: Remove RDMA protocol handling
On Wed, Jun 05, 2024 at 08:48:28PM +0000, Dr. David Alan Gilbert wrote:
> > > I just noticed this thread; some random notes from a somewhat
> > > fragmented memory of this:
> > >
> > > a) Long long ago, I also tried rsocket;
> > >
https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg02040.html
> > > as I remember the library was quite flaky at the time.
> >
> > Hmm interesting. There also looks like a thread doing rpoll().
>
> Yeh, I can't actually remember much more about what I did back then!
Heh, that's understandable and fair. :)
> > I hope Lei and his team has tested >4G mem, otherwise definitely
> > worth checking. Lei also mentioned there're rsocket bugs they found
> > in the cover letter, but not sure what's that about.
>
> It would probably be a good idea to keep track of what bugs are in
> flight with it, and try it on a few RDMA cards to see what problems
> get triggered.
> I think I reported a few at the time, but I gave up after feeling it
> was getting very hacky.
Agreed. Maybe we can have a list of that in the cover letter or even QEMU's
migration/rmda doc page.
Lei, if you think that makes sense please do so in your upcoming posts.
There'll need to have a list of things you encountered in the kernel driver and
it'll be even better if there're further links to read on each problem.
OK, no problem. There are two bugs:
Bug 1:
https://github.com/linux-rdma/rdma-core/commit/23985e25aebb559b761872313f...
his commit introduces a bug that causes QEMU suspension.
When the timeout parameter of the rpoll is not -1 or 0, the program is suspended
occasionally.
Problem analysis:
During the first rpoll,
In line 3297, rs_poll_enter () performs pollcnt++. In this case, the value of pollcnt is
1.
In line 3302, timeout expires and the function exits. Note that rs_poll_exit () is not
--pollcnt here.
In this case, the value of pollcnt is 1.
During the second rpoll, pollcnt++ is performed in line 3297 rs_poll_enter (). In this
case, the value of pollcnt is 2.
If no timeout expires and the poll return value is greater than 0, the rs_poll_stop ()
function is executed. Because the if (--pollcnt) condition is false, suspendpoll = 1 is
executed.
Go back to the do while loop inside rpoll, again rs_poll_enter () now if (suspendpoll)
condition is true, execute pthread_yield (); and return -EBUSY, Then, the do while loop in
the rpoll is returned. Because the if (rs_poll_enter ()) condition is true, the
rs_poll_enter () function is executed again after the continue operation. As a result, the
program is suspended.
Root cause: In line 3302, rs_poll_exit () is not executed before the timeout expires
function exits.
Bug 2:
In rsocket.c, there is a receive queue int accept_queue[2] implemented by socketpair. The
listen_svc thread in rsocket.c is responsible for receiving connections and writing them
to the accept_queue[1]. When raccept () is called, a connection is received from
accept_queue[0].
In the test case, qio_channel_wait(QIO_CHANNEL(lioc), G_IO_IN); waits for a readable event
(waiting for a connection), rpoll () checks if accept_queue[0] has a readable event,
However, this poll does not poll accept_queue[0]. After the timeout expires, rpoll ()
obtains the readable event of accept_queue[0] from rs_poll_arm again.
Impaction:
The accept operation can be performed only after 5000 ms. Of course, we can shorten this
time by echoing the millisecond time > /etc/rdma/rsocket/wake_up_interval.
Regards,
-Gonglei
> > >
> > > e) Someone made a good suggestion (sorry can't remember who) -
that the
> > > RDMA migration structure was the wrong way around - it should
be the
> > > destination which initiates an RDMA read, rather than the source
> > > doing a write; then things might become a LOT simpler; you just
need
> > > to send page ranges to the destination and it can pull it.
> > > That might work nicely for postcopy.
> >
> > I'm not sure whether it'll still be a problem if rdma recv side is
> > based on zero-copy. It would be a matter of whether atomicity can
> > be guaranteed so that we don't want the guest vcpus to see a
> > partially copied page during on-flight DMAs. UFFDIO_COPY (or
> > friend) is currently the only solution for that.
>
> Yes, but even ignoring that (and the UFFDIO_CONTINUE idea you
> mention), if the destination can issue an RDMA read itself, it doesn't
> need to send messages to the source to ask for a page fetch; it just
> goes and grabs it itself, that's got to be good for latency.
Oh, that's pretty internal stuff of rdma to me and beyond my knowledge..
but from what I can tell it sounds very reasonable indeed!
Thanks!
--
Peter Xu