7:52 p.m.
Hi All,
Matt encountered the following issue when using 1GB huge pages with libvirt
This problem turned out to be entirely my fault because I didn't
round
the VM's memory size to a 1G multiple, and the kernel tried to split
the VMA at the end of the region, triggering this code in the kernel
do_mbind() path,
static int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
unsigned long addr, int new_below)
{
struct vm_area_struct *new;
int err;
if (is_vm_hugetlb_page(vma) && (addr &
~(huge_page_mask(hstate_vma(vma)))))
return -EINVAL;
I have no idea how a less fortunate developer without access to MM
experts would have figured this out. It's a shame virsh didn't error
out when I initially setup 1G hugepages with an incompatible VM memory
size.
What do folks think about improving libvirt to warn or error when using a VM
memory size that is not compatible with the host hugepage configuration?
Regards,
Jim
4:23 a.m.
On 02/13/2017 02:52 AM, Jim Fehlig wrote:
Hi All,
Matt encountered the following issue when using 1GB huge pages with libvirt
> This problem turned out to be entirely my fault because I didn't round
> the VM's memory size to a 1G multiple, and the kernel tried to split
> the VMA at the end of the region, triggering this code in the kernel
> do_mbind() path,
>
> static int __split_vma(struct mm_struct *mm, struct vm_area_struct
> *vma,
> unsigned long addr, int new_below)
> {
> struct vm_area_struct *new;
> int err;
>
> if (is_vm_hugetlb_page(vma) && (addr &
>
> ~(huge_page_mask(hstate_vma(vma)))))
> return -EINVAL;
>
> I have no idea how a less fortunate developer without access to MM
> experts would have figured this out. It's a shame virsh didn't error
> out when I initially setup 1G hugepages with an incompatible VM memory
> size.
What do folks think about improving libvirt to warn or error when using
a VM memory size that is not compatible with the host hugepage
configuration?
Jim,
what is the actual problem? I've tried to reproduce this by running vm
with 3.5G RAM backed by 1GB huge pages and the guest runs just fine. I
mean kvm guest. This is because at the cmd line level there is aligned
value:
-m size=4194304k,slots=16,maxmem=8388608k
This is result of qemuDomainAlignMemorySizes(). So perhaps there's a bug
somewhere in the function?
Michal
8:26 a.m.
On Tue, 21 Feb, at 11:23:52AM, Michal Privoznik wrote:
Jim,
what is the actual problem? I've tried to reproduce this by running vm
with 3.5G RAM backed by 1GB huge pages and the guest runs just fine. I
mean kvm guest. This is because at the cmd line level there is aligned
value:
-m size=4194304k,slots=16,maxmem=8388608k
This is result of qemuDomainAlignMemorySizes(). So perhaps there's a bug
somewhere in the function?
Quite possibly. Some memory values work fine and appear to be rounded
to the next gigabyte boundary. One KiB values that fails for me is,
<memory unit='KiB'>52428801</memory>
<currentMemory unit='KiB'>52428801</currentMemory>
Which results in qemu-kvm ... -m 51201
9:50 a.m.
On 02/23/2017 03:26 PM, Matt Fleming wrote:
On Tue, 21 Feb, at 11:23:52AM, Michal Privoznik wrote:
>
> Jim,
>
> what is the actual problem? I've tried to reproduce this by running vm
> with 3.5G RAM backed by 1GB huge pages and the guest runs just fine. I
> mean kvm guest. This is because at the cmd line level there is aligned
> value:
>
> -m size=4194304k,slots=16,maxmem=8388608k
>
> This is result of qemuDomainAlignMemorySizes(). So perhaps there's a bug
> somewhere in the function?
Quite possibly. Some memory values work fine and appear to be rounded
to the next gigabyte boundary. One KiB values that fails for me is,
<memory unit='KiB'>52428801</memory>
<currentMemory unit='KiB'>52428801</currentMemory>
Which results in qemu-kvm ... -m 51201
I currently don't have access to a host with 50+GB of RAM, so I start small:
<memory unit='KiB'>3145739</memory>
<currentMemory unit='KiB'>3145739</currentMemory>
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB'/>
</hugepages>
</memoryBacking>
-m 3073
And this works just fine. One thing that I've noticed is that we don't
take into account sizes of huge pages in qemuDomainAlignMemorySizes().
But then again - what is the scenario you're seeing? What's the error
message?
Michal
5:38 p.m.
On 02/23/2017 08:50 AM, Michal Privoznik wrote:
On 02/23/2017 03:26 PM, Matt Fleming wrote:
> On Tue, 21 Feb, at 11:23:52AM, Michal Privoznik wrote:
>>
>> Jim,
>>
>> what is the actual problem? I've tried to reproduce this by running vm
>> with 3.5G RAM backed by 1GB huge pages and the guest runs just fine. I
>> mean kvm guest. This is because at the cmd line level there is aligned
>> value:
>>
>> -m size=4194304k,slots=16,maxmem=8388608k
>>
>> This is result of qemuDomainAlignMemorySizes(). So perhaps there's a bug
>> somewhere in the function?
>
> Quite possibly. Some memory values work fine and appear to be rounded
> to the next gigabyte boundary. One KiB values that fails for me is,
>
> <memory unit='KiB'>52428801</memory>
> <currentMemory unit='KiB'>52428801</currentMemory>
>
> Which results in qemu-kvm ... -m 51201
I currently don't have access to a host with 50+GB of RAM, so I start small:
<memory unit='KiB'>3145739</memory>
<currentMemory unit='KiB'>3145739</currentMemory>
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB'/>
</hugepages>
</memoryBacking>
-m 3073
And this works just fine. One thing that I've noticed is that we don't
take into account sizes of huge pages in qemuDomainAlignMemorySizes().
But then again - what is the scenario you're seeing? What's the error
message?
Sorry for the delay. I was having problems reproducing it until Matt reminded me
that some vNUMA was needed in the mix. E.g. config containing
<memory unit='KiB'>10000000</memory>
<currentMemory unit='KiB'>10000000</currentMemory>
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB'/>
</hugepages>
</memoryBacking>
<cpu mode='host-passthrough'>
<topology sockets='2' cores='20' threads='2'/>
<numa>
<cell id='0' cpus='0-39' memory='5000000'
unit='KiB'/>
<cell id='1' cpus='40-79' memory='5000000'
unit='KiB'/>
</numa>
</cpu>
fails with
2017-03-14T21:50:01.228693Z qemu-system-x86_64: -object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,size=5120196608,host-nodes=0-1,policy=bind:
cannot bind memory to host NUMA nodes: Invalid argument
Changing the config by s/10000000/10485760/ and s/5000000/5242880/ works. It was
only then that I remembered the machine was running libvirt 2.0.0 :-/. I updated
to 3.1.0 and the original, unaligned config works. So there are some fixes in
the alignment code in the meanwhile. I'll test some of the other problematic
configurations, which in some cases resulted in different qemu errors, and
report back if I find any problems with current code.
Regards,
Jim
2809
days inactive
2838
days old
4 comments
3 participants
participants (3)
-
Jim Fehlig -
Matt Fleming -
Michal Privoznik