Re: [PATCH v4 for v7.6.0 00/14] Introduce virtio-mem <memory/> model

Thanks for looking into this. It's a fairly long list, so it's understandable that it took a while. :) Yeah, but it can certainly be considered as future work as well. So it could be backed by a file residing on a filesystem that supports sparse files (shmem, hugetlbfs, ext4, ...) -- IOW anything modern :) It's supposed to work, but if it makes your life easier, we can consider supporting other file sources future work. So "prealloc=on" support for virtio-mem is unfortunately not part of QEMU yet (only "reserve=off" for memory backends). As you correctly state, until that is in place, huge pages cannot be used in a safe way with virtio-mem, which is why they are not officially supported yet by virtio-mem. The idea is to specify "prealloc=on" on the virtio-mem device level once supported, instead of on the memory backend level. So virtio-mem will preallocate the relevant memory before actually giving new block to the guest via virtio-mem, not when creating the memory backend. If preallocation fails at that point, no new blocks are given to the guest and we won't get killed. Think of it like this: you defer preallocation to the point where you actually use the memory and handle preallocation errors still in a safe way. More details are below. Assume you want a 1TB virtio-mem device backed by huge pages but initially only expose 1GB to the VM. When setting prealloc=on on the memory backend, we will preallocate 1TB of huge pages when starting QEMU to discard the memory immediately again within virtio-mem startup code (first thing it does is make sure there is no memory backing at all, meaning the memory backend is completely "empty"). We end up with no preallocated memory and temporarily having allocated 1 TB. When setting "prealloc=on" (once supported) on the virtio-mem device instead, we'll preallocate memory dynamically as we hand it to the VM -- so initially only 1GB. Assume we want to give the VM an additional 16GB via that virtio-mem device. virtio-mem will dynamically try preallocating the memory before giving the guest 16GB. Assume only 8GB could be preallocated, then the VM will only get additional 8GB and we won't crash. [...] It's difficult the way especially huge pages work. Say you mmap() 1TB of huge pages. Linux will "reserve" 1TB of huge pages and fail mmap() if it can't. BUT it will not preallocate huge pages yet, they are only accounted for in the OS as reserved for this mapping. The idea is that you cannot really overcommit huge pages in the traditional sense, so the reservation mechanism was implemented to make it harder for user space to do something stupid. BUT we still need preallocation because the whole "huge page reservation" code is broken and not NUMA aware! This, however, breaks the idea of virtio-mem, where you want to dynamically decide how much memory you actually give to a VM. If you reserve all huge pages of the memory backend upfront, they cannot be used for anything else in the meantime and you can just stop using virtio-mem and use a large DIMM instead. In the end, what we want in virtio-mem with huge pages in the future is: * reserve=off for the memory backend: don't reserve any huge pages by in the OS, we'll be preallocating instead. * prealloc=on for the virtio-mem device: preallocate memory dynamically when really about to be used by the VM and fail in a safe way if preallcoation fails. In addition to that, "reserve=off" can be useful with virtio-mem also when backed by ordinary system RAM where we don't use preallocation, just due to the way some memory overcommit modes work. But that's also stuff for the future to optimize and you don't have to bother about that just now. :) -- Thanks, David / dhildenb