On 02/04/2014 10:56 PM, Jiri Denemark wrote:
On Mon, Jan 13, 2014 at 14:28:12 +0800, mrhines@linux.vnet.ibm.com wrote:
From: "Michael R. Hines" <mrhines@us.ibm.com>
RDMA Live migration requires registering memory with the hardware, Hmm, I forgot to ask when I was reviewing the previous patch but does any of this RDMA migration functionality require special privileges or is any unprivileged process able to use RDMA?
No, it does not require any special privileges (just like HPC programs), but it does access a bunch of special device files (unprivleged): I believe someone recommended that we had the list of those device files to the default list of allowed devices that libvirt is already maintaining..... I'll include them in the next patch....
+{ + qemuDomainObjPrivatePtr priv = vm->privateData; + int ret; + + if (qemuDomainObjEnterMonitorAsync(driver, vm, job) < 0) + return -1; + + ret = qemuMonitorGetMigrationCapability( + priv->mon, + QEMU_MONITOR_MIGRATION_CAPS_RDMA_PIN_ALL); Is this capability always present when QEMU supports RDMA migration? If so, it could be used when we check if QEMU supports RDMA migration.
See my comment earlier in the thread....... Yes, it's present, but it still does not guarantee that QEMU supports it if RDMA was compiled out - only the version number is a (minimal) guarantee, and even then the hardware can still throw an error if RDMA itself is not supported. I'm still not seeing what's wrong with depending on the version number since other features are also depending on the version number. Are you guys suggesting that we stop depending on version altogether for *all* QEMU features?
+ unsigned long long memKB = vm->def->mem.hard_limit ? + vm->def->mem.hard_limit : + vm->def->mem.max_balloon + 1024 * 1024; + virProcessSetMaxMemLock(vm->pid, memKB * 3); Apart from weird indentation of the virProcessSetMaxMemLock line, there are several issues here:
- this code should be moved inside qemuMigrationSetPinAll and done only if VIR_MIGRATE_RDMA_PIN_ALL flag was used.
- virProcessSetMaxMemLock wants the value in bytes, thus memKB should rather by multiplied by 1024.
- what memory is going to be mlock()ed with rdma-pin-all? Is it going to be all memory allocated by QEMU or just the domain's memory? If it's all QEMU memory, we already painfully found out it's impossible to automatically compute how much memory QEMU consumes in addition to the configured domain's memory and I think a better approach would be to just migration with rdma-pin-all unless hard_limit is specified. (Acknowledged for the first two comments).
Regarding your 3rd part: That's why I multiplied the number by 3, the RDMA code *must* lock or the whole thing falls apart, so we have to make "some kind" of reasonable assumption on how much to set the lock limit to. I'm willing to go even higher than 3 times the limit, if nobody objects, but we have to pick some kind of limit......somehow. Comments? - Michael