[libvirt] storage volume download/upload
As i see in some cases, for example in case of lvm storage pool libvirt spawns libvirt_iohelper to upload or download data to volume. But in case of big data contains zeroes this is wast bw of network. May be we can use gzip or lz4 for compression stream? How the best do this not breaking exiting software? May be add new parameter to libvirt_iohelper with compression method and compression level? In my cases of 20-160 Gb data and used only 4-5 Gb this provide major benefit.... Thanks! -- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru jabber: vase@selfip.ru
No suggestions ? 2015-05-05 16:18 GMT+03:00 Vasiliy Tolstov <v.tolstov@selfip.ru>:
As i see in some cases, for example in case of lvm storage pool libvirt spawns libvirt_iohelper to upload or download data to volume. But in case of big data contains zeroes this is wast bw of network. May be we can use gzip or lz4 for compression stream? How the best do this not breaking exiting software? May be add new parameter to libvirt_iohelper with compression method and compression level? In my cases of 20-160 Gb data and used only 4-5 Gb this provide major benefit.... Thanks!
-- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru jabber: vase@selfip.ru
-- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru jabber: vase@selfip.ru
On 05.05.2015 15:18, Vasiliy Tolstov wrote:
As i see in some cases, for example in case of lvm storage pool libvirt spawns libvirt_iohelper to upload or download data to volume. But in case of big data contains zeroes this is wast bw of network. May be we can use gzip or lz4 for compression stream? How the best do this not breaking exiting software? May be add new parameter to libvirt_iohelper with compression method and compression level? In my cases of 20-160 Gb data and used only 4-5 Gb this provide major benefit.... Thanks!
The main benefit of using iohelper is that we have aligned reads from disks and thus can use O_DIRECT flag. So if anything, any compression must be put in between iohelper and the daemon which then resends data to clients. Unfortunately, we don't have a field in the API signature that can be used to select desired algorithm. We have @flags which could be used to enable compression. Which algorithm to use can be set in a config file (libvirtd.conf seems most suitable). Michal
2015-05-07 17:06 GMT+03:00 Michal Privoznik <mprivozn@redhat.com>:
The main benefit of using iohelper is that we have aligned reads from disks and thus can use O_DIRECT flag. So if anything, any compression must be put in between iohelper and the daemon which then resends data to clients.
but if iohelper after read data from disk compress it, does it suitable?
Unfortunately, we don't have a field in the API signature that can be used to select desired algorithm. We have @flags which could be used to enable compression. Which algorithm to use can be set in a config file (libvirtd.conf seems most suitable).
May be encode in flags compressor ? So we don't need additional entry in config file and without needed flag iohelper does not compress at all ? -- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru jabber: vase@selfip.ru
On 07.05.2015 16:14, Vasiliy Tolstov wrote:
2015-05-07 17:06 GMT+03:00 Michal Privoznik <mprivozn@redhat.com>:
The main benefit of using iohelper is that we have aligned reads from disks and thus can use O_DIRECT flag. So if anything, any compression must be put in between iohelper and the daemon which then resends data to clients.
but if iohelper after read data from disk compress it, does it suitable?
Yes, it's suitable.
Unfortunately, we don't have a field in the API signature that can be used to select desired algorithm. We have @flags which could be used to enable compression. Which algorithm to use can be set in a config file (libvirtd.conf seems most suitable).
May be encode in flags compressor ? So we don't need additional entry in config file and without needed flag iohelper does not compress at all ?
This is what I'm trying to avoid: having a flag per every compression algorithm out there in wildness. Flags are meant to slightly change behaviour of a function, not to encode an enum. It's a waste too: to encode say 4 algorithms as flags you need 4 bits, while in reality you'd need only 2. The more algorithms we introduce the more bits from @flags are consumed, leaving less space for something more useful. Also, I guess algorithms will not be chained (it does not make much sense either), so flags would need to be mutually exclusive. Therefore my initial approach, with just a single flag to enable compression, and leave algorithm to be selected somewhere else. Michal
participants (2)
-
Michal Privoznik -
Vasiliy Tolstov