First off all thank you for your answer,

I couldn't figured how to start virtual machine with increased MEMLOCK,

tried to add into /etc/security/limits.d

qemu soft memlock 3221225

qemu hard memlock 3221225

so max locked-in-memory will be 3G, but it didn't worked.

still has MEMLOCK of 60kb per each VM.

Maybe you can spot what I'm doing wrong?

On Tue, Feb 9, 2016 at 5:16 PM, Michael R. Hines <michael@hinespot.com> wrote:

Hi Roy,

On 02/09/2016 03:57 AM, Roy Shterman wrote:

Hi,

I tried to understand the rdma-migration in qemu code and i have two questions about it:

1. I'm working with qemu-kvm using libvirt and i'm getting

MEMLOCK max locked-in-memory address space 65536 65536 bytes

in qemu process so I don't understand how can you use rdma-pin-all with such low MEMLOCK.

I found a solution in libvirt to lock all vm memory in advance and to enlarge MEMLOCK.
It uses memoryBacking locking and memory tuning hard_limit of vm memory but I couldn't find a usage of this in rdma-migration code.

You're absolutey right, the RDMA migration code itself doesn't set this lock limit explicitly because there are system-wide restrictions in both appArmour,
/etc/security, as well as SELINUX that restrict applications from arbitrarily setting their maximum memory lock limits.

The other problem is CGROUPS: If someone sets a cgroup control for maximum memory and forgets about that mlock() limits, then
there will be a conflict.

So, libvirt must have a policy to deal with all of these possibilities, not just handle a special case for RDMA migration.

The only way "simple" way (without patching the problems above) to apply a higher lock limit to QEMU is to set the ulimit for libvirt
(or for QEMU if starting QEMU manually) in your environment or the command line with $ ulimit # before attempting the migration,
then the RDMA subsystem will be able to lock the memory successfully.

The other option is to use /etc/security/limits.conf and set the option for a specific libvirt process user and make sure your libvirt/qemu
are not running as root.

QEMU itself also has a "mlock" option built into the command line, but it also suffers from the same problem --- you have to find
a way (currently) to increase the limit before using the option.

2. Do you have any comparison of IOPS and bandwidth between TCP migration and rdma migration?

Yes, lots of comparisons.

http://wiki.qemu.org/Features/RDMALiveMigration
http://www.canturkisci.com/ETC/papers/IBMJRD2011/preprint.pdf

Regards,
Roy