Re: [libvirt] question about rdma migration

Is the QEMU process (after startup) actually running as the QEMU userid ? /* * Michael R. Hines * Platform Engineer, DigitalOcean. */ On 02/19/2016 02:43 PM, Roy Shterman wrote: First off all thank you for your answer, I couldn't figured how to start virtual machine with increased MEMLOCK, tried to add into /etc/security/limits.d qemu soft memlock 3221225 qemu hard memlock 3221225 so max locked-in-memory will be 3G, but it didn't worked. still has MEMLOCK of 60kb per each VM. Maybe you can spot what I'm doing wrong? On Tue, Feb 9, 2016 at 5:16 PM, Michael R. Hines < <michael(a)hinespot.com&gt; michael(a)hinespot.com&gt; wrote: > Hi Roy, > > On 02/09/2016 03:57 AM, Roy Shterman wrote: > >> Hi, >> >> I tried to understand the rdma-migration in qemu code and i have two >> questions about it: >> >> 1. I'm working with qemu-kvm using libvirt and i'm getting >> >> MEMLOCK max locked-in-memory address space 65536 65536 bytes >> >> in qemu process so I don't understand how can you use rdma-pin-all with >> such low MEMLOCK. >> >> I found a solution in libvirt to lock all vm memory in advance and to >> enlarge MEMLOCK. >> It uses memoryBacking locking and memory tuning hard_limit of vm memory >> but I couldn't find a usage of this in rdma-migration code. >> >> > You're absolutey right, the RDMA migration code itself doesn't set this > lock limit explicitly because there are system-wide restrictions in both > appArmour, > /etc/security, as well as SELINUX that restrict applications from > arbitrarily setting their maximum memory lock limits. > > The other problem is CGROUPS: If someone sets a cgroup control for > maximum memory and forgets about that mlock() limits, then > there will be a conflict. > > So, libvirt must have a policy to deal with all of these possibilities, > not just handle a special case for RDMA migration. > > The only way "simple" way (without patching the problems above) to apply > a higher lock limit to QEMU is to set the ulimit for libvirt > (or for QEMU if starting QEMU manually) in your environment or the > command line with $ ulimit # before attempting the migration, > then the RDMA subsystem will be able to lock the memory successfully. > > The other option is to use /etc/security/limits.conf and set the option > for a specific libvirt process user and make sure your libvirt/qemu > are not running as root. > > QEMU itself also has a "mlock" option built into the command line, but it > also suffers from the same problem --- you have to find > a way (currently) to increase the limit before using the option. > > 2. Do you have any comparison of IOPS and bandwidth between TCP migration >> and rdma migration? >> >> Yes, lots of comparisons. > > http://wiki.qemu.org/Features/RDMALiveMigration > http://www.canturkisci.com/ETC/papers/IBMJRD2011/preprint.pdf > > > Regards, >> Roy >> >> >> > -- libvir-list mailing listlibvir-list@redhat.comhttps://www.redhat.com/mailman/listinfo/libvir-list