[libvirt-users] libvirtd does not update VM .xml configuration on filesystem after virsh blockcommit
Hi, Recently We've upgraded some KVM hosts from Fedora 29 to Fedora 30 and now experience broken VM configurations on filesystem after virsh blockcommit. Commands "virsh dumpxml ..." and "virsh dumpxml --inactive ..." is showing diffrent configuration than the one on filesystem. In case of restart libvirtd or system reboot, there are broken VM xml configurations on filesystem. Everything is OK on Fedora 29 KVM hosts! 0. XML configurations before snapshot is taken (all good, nothing found) [root@server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml| grep BACK [root@server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml| grep backingStore [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep BACK [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep backingStore [root@server1 ~]# /usr/bin/virsh --quiet dumpxml --inactive somedomain.com.ncloud|grep BACK [root@server1 ~]# /usr/bin/virsh --quiet dumpxml --inactive somedomain.com.ncloud|grep backingStore 1. When VM snapshot is taken with (all OK) /usr/bin/virsh --quiet snapshot-create-as --domain somedomain.com.ncloud .... there is a change of configuration, active, inactive and filesystem: - active (virsh dumpxml) [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep BACK <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW'/> [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep backingStore <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> - inactive (virsh dumpxml --inactive) [root@server1 ~]# /usr/bin/virsh --quiet dumpxml --inactive somedomain.com.ncloud|grep BACK <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW'/> [root@server1 ~]# /usr/bin/virsh --quiet dumpxml --inactive somedomain.com.ncloud|grep backingStore <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> - XML configuration on filesystem has changed [root@server1 ~]# ls -al /etc/libvirt/qemu/somedomain.com.ncloud.xml -rw-------. 1 root root 6260 Jun 18 23:00 /etc/libvirt/qemu/somedomain.com.ncloud.xml [root@server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml |grep BACK <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW'/> [root@server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml |grep backingStore <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> 2. When VM backup is done, data merged with virsh backcommit ..." and snapshot deleted (NOT OK!!!) ... /usr/bin/virsh --quiet blockcommit somedomain.com.ncloud sdd --active --pivot /usr/bin/virsh --quiet snapshot-delete --domain somedomain.com.ncloud somedomain.com.ncloud-SNAPSHOT --metadata there is a following state of VM configurations: - active (virsh dumpxml), [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep BACK ;;; OK [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep backingStore ;;; why is there empty backingStore left ??? <backingStore/> <backingStore/> <backingStore/> <backingStore/> - inactive [root@server1 qemu]# virsh dumpxml --inactive somedomain.com.ncloud |grep BACK ;;; OK [root@server1 qemu]# virsh dumpxml --inactive somedomain.com.ncloud |grep backingStore ;;; OK - XML on filesystem (.xml file on filesystem has not changed/reverted since snapshot has been taken - NOT OK!!!!, should be cleared of snapshot source file and backingStore) [root@server1 ~]# ls -al /etc/libvirt/qemu/somedomain.com.ncloud.xml -rw-------. 1 root root 6260 Jun 18 23:00 /etc/libvirt/qemu/somedomain.com.ncloud.xml [root@server1 qemu]# cat somedomain.com.ns2.xml |grep BACK <source file='/Virtualization/linux/somedomain.com/somedomain.com.ns2.qcow2-BACKUPING_NOW'/> [root@server1 qemu]# cat somedomain.com.ns2.xml |grep backingStore <backingStore type='file' index='1'> </backingStore> ############################################################################################################################################################################## - Fedora 29 has libvirt 4.7.0 and qemu 3.0.1: [root@solaris1 ~]# rpm -qa |grep libvirt-daemon-kvm libvirt-daemon-kvm-4.7.0-3.fc29.x86_64 [root@solaris1 ~]# rpm -qa |grep qemu-system-x86 qemu-system-x86-core-3.0.1-3.fc29.x86_64 qemu-system-x86-3.0.1-3.fc29.x86_64 - Fedora 30 has libvirt 5.1.0 and qemu 3.1.0: [root@server1 ~]# rpm -qa |grep libvirt-daemon-kvm libvirt-daemon-kvm-5.1.0-8.fc30.x86_64 [root@server1 ~]# rpm -qa |grep qemu-system-x86 qemu-system-x86-3.1.0-8.fc30.x86_64 qemu-system-x86-core-3.1.0-8.fc30.x86_64 For every VM from "virsh list" we do following steps (in script) for VM backup: /usr/bin/virsh --quiet domblklist somedomain.com.ncloud /usr/bin/virsh --quiet dumpxml --inactive somedomain.com.ncloud > /Backuping/VMs/Daily/somedomain.com.ncloud.xml /usr/bin/virsh --quiet snapshot-create-as --domain somedomain.com.ncloud somedomain.com.ncloud-SNAPSHOT --diskspec sda,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW --diskspec sdb,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW --diskspec sdc,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW --diskspec sdd,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW --disk-only --atomic --quiesce /usr/bin/virsh --quiet snapshot-list somedomain.com.ncloud /usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2 somedomain.com.ncloud.qcow2 /usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2 somedomain.com.ncloud-swap.qcow2 /usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2 somedomain.com.ncloud-data1.qcow2 /usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2 somedomain.com.ncloud-data2.qcow2 /usr/bin/virsh --quiet blockcommit somedomain.com.ncloud sda --active --pivot /usr/bin/virsh --quiet blockcommit somedomain.com.ncloud sdb --active --pivot /usr/bin/virsh --quiet blockcommit somedomain.com.ncloud sdc --active --pivot /usr/bin/virsh --quiet blockcommit somedomain.com.ncloud sdd --active --pivot /usr/bin/virsh --quiet snapshot-delete --domain somedomain.com.ncloud somedomain.com.ncloud-SNAPSHOT --metadata /usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW" /usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW" /usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW" /usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW" /usr/bin/pigz --best --rsyncable somedomain.com.ncloud.qcow2 /usr/bin/pigz --best --rsyncable somedomain.com.ncloud-swap.qcow2 /usr/bin/pigz --best --rsyncable somedomain.com.ncloud-data1.qcow2 /usr/bin/pigz --best --rsyncable somedomain.com.ncloud-data2.qcow2 There is no error on script (commands) execution: [root@server1 ~]# /Backuping/bin/simple_KVM_backup.pl --depth 7 --path Daily --hosting_server server1.somedomain.us ###Backuping VM somedomain.com.ncloud: + Moving somedomain.com.ncloud.xml.06 to somedomain.com.ncloud.xml.07. + Moving somedomain.com.ncloud.qcow2.gz.06 to somedomain.com.ncloud.qcow2.gz.07. ... + Disk snapshot somedomain.com.ncloud-SNAPSHOT for somedomain.com.ncloud ... Domain snapshot somedomain.com.ncloud-SNAPSHOT created Name Creation Time State ----------------------------------------------------------------------- somedomain.com.ncloud-SNAPSHOT 2019-06-18 23:00:04 +0200 disk-snapshot Done! + Time scp start : 2019-06-18 23:00:05 somedomain.com.ncloud.qcow2 100% 20GB 401.3MB/s 00:51 Done! somedomain.com.ncloud-swap.qcow2 100% 4097MB 483.5MB/s 00:08 Done! somedomain.com.ncloud-data1.qcow2 100% 100GB 400.7MB/s 04:15 Done! somedomain.com.ncloud-data2.qcow2 100% 100GB 440.6MB/s 03:52 Done! + Time scp end : 2019-06-18 23:09:14 + Time blockcommit start : 2019-06-18 23:09:14 Successfully pivoted Successfully pivoted Successfully pivoted Successfully pivoted + Time blockcommit end : 2019-06-18 23:09:16 + Snapshot delete ... Domain snapshot somedomain.com.ncloud-SNAPSHOT deleted Done! + Delete snapshot files ... Done! + Time compress start : 2019-06-18 23:09:17 Regards, saso
Hi, Recently We've upgraded some KVM hosts from Fedora 29 to Fedora 30 and now experience broken VM configurations on filesystem after virsh blockcommit. Commands "virsh dumpxml ..." and "virsh dumpxml --inactive ..." is showing diffrent configuration than the one on filesystem. In case of restart libvirtd or system reboot, there are broken VM xml configurations on filesystem. Everything is OK on Fedora 29 KVM hosts! 0. XML configurations before snapshot is taken (all good, nothing found) [root@server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml| grep BACK [root@server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml| grep backingStore [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep BACK [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep backingStore [root@server1 ~]# /usr/bin/virsh --quiet dumpxml --inactive somedomain.com.ncloud|grep BACK [root@server1 ~]# /usr/bin/virsh --quiet dumpxml --inactive somedomain.com.ncloud|grep backingStore 1. When VM snapshot is taken with (all OK) /usr/bin/virsh --quiet snapshot-create-as --domain somedomain.com.ncloud .... there is a change of configuration, active, inactive and filesystem: - active (virsh dumpxml) [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep BACK <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW'/> [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep backingStore <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> - inactive (virsh dumpxml --inactive) [root@server1 ~]# /usr/bin/virsh --quiet dumpxml --inactive somedomain.com.ncloud|grep BACK <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW'/> [root@server1 ~]# /usr/bin/virsh --quiet dumpxml --inactive somedomain.com.ncloud|grep backingStore <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> - XML configuration on filesystem has changed [root@server1 ~]# ls -al /etc/libvirt/qemu/somedomain.com.ncloud.xml -rw-------. 1 root root 6260 Jun 18 23:00 /etc/libvirt/qemu/somedomain.com.ncloud.xml [root@server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml |grep BACK <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW'/> <source file='/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW'/> [root@server1 ~]# cat /etc/libvirt/qemu/somedomain.com.ncloud.xml |grep backingStore <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> <backingStore type='file'> </backingStore> 2. When VM backup is done, data merged with virsh backcommit ..." and snapshot deleted (NOT OK!!!) ... /usr/bin/virsh --quiet blockcommit somedomain.com.ncloud sdd --active --pivot /usr/bin/virsh --quiet snapshot-delete --domain somedomain.com.ncloud somedomain.com.ncloud-SNAPSHOT --metadata there is a following state of VM configurations: - active (virsh dumpxml), [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep BACK ;;; OK [root@server1 ~]# /usr/bin/virsh --quiet dumpxml somedomain.com.ncloud|grep backingStore ;;; why is there empty backingStore left ??? <backingStore/> <backingStore/> <backingStore/> <backingStore/> - inactive [root@server1 qemu]# virsh dumpxml --inactive somedomain.com.ncloud |grep BACK ;;; OK [root@server1 qemu]# virsh dumpxml --inactive somedomain.com.ncloud |grep backingStore ;;; OK - XML on filesystem (.xml file on filesystem has not changed/reverted since snapshot has been taken - NOT OK!!!!, should be cleared of snapshot source file and backingStore) [root@server1 ~]# ls -al /etc/libvirt/qemu/somedomain.com.ncloud.xml -rw-------. 1 root root 6260 Jun 18 23:00 /etc/libvirt/qemu/somedomain.com.ncloud.xml [root@server1 qemu]# cat somedomain.com.ns2.xml |grep BACK <source file='/Virtualization/linux/somedomain.com/somedomain.com.ns2.qcow2-BACKUPING_NOW'/> [root@server1 qemu]# cat somedomain.com.ns2.xml |grep backingStore <backingStore type='file' index='1'> </backingStore> ############################################################################################################################################################################## - Fedora 29 has libvirt 4.7.0 and qemu 3.0.1: [root@solaris1 ~]# rpm -qa |grep libvirt-daemon-kvm libvirt-daemon-kvm-4.7.0-3.fc29.x86_64 [root@solaris1 ~]# rpm -qa |grep qemu-system-x86 qemu-system-x86-core-3.0.1-3.fc29.x86_64 qemu-system-x86-3.0.1-3.fc29.x86_64 - Fedora 30 has libvirt 5.1.0 and qemu 3.1.0: [root@server1 ~]# rpm -qa |grep libvirt-daemon-kvm libvirt-daemon-kvm-5.1.0-8.fc30.x86_64 [root@server1 ~]# rpm -qa |grep qemu-system-x86 qemu-system-x86-3.1.0-8.fc30.x86_64 qemu-system-x86-core-3.1.0-8.fc30.x86_64 For every VM from "virsh list" we do following steps (in script) for VM backup: /usr/bin/virsh --quiet domblklist somedomain.com.ncloud /usr/bin/virsh --quiet dumpxml --inactive somedomain.com.ncloud > /Backuping/VMs/Daily/somedomain.com.ncloud.xml /usr/bin/virsh --quiet snapshot-create-as --domain somedomain.com.ncloud somedomain.com.ncloud-SNAPSHOT --diskspec sda,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW --diskspec sdb,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW --diskspec sdc,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW --diskspec sdd,file=/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW --disk-only --atomic --quiesce /usr/bin/virsh --quiet snapshot-list somedomain.com.ncloud /usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2 somedomain.com.ncloud.qcow2 /usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2 somedomain.com.ncloud-swap.qcow2 /usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2 somedomain.com.ncloud-data1.qcow2 /usr/bin/scp -p server1.somedomain.us:/Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2 somedomain.com.ncloud-data2.qcow2 /usr/bin/virsh --quiet blockcommit somedomain.com.ncloud sda --active --pivot /usr/bin/virsh --quiet blockcommit somedomain.com.ncloud sdb --active --pivot /usr/bin/virsh --quiet blockcommit somedomain.com.ncloud sdc --active --pivot /usr/bin/virsh --quiet blockcommit somedomain.com.ncloud sdd --active --pivot /usr/bin/virsh --quiet snapshot-delete --domain somedomain.com.ncloud somedomain.com.ncloud-SNAPSHOT --metadata /usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud.qcow2-BACKUPING_NOW" /usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud-swap.qcow2-BACKUPING_NOW" /usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud-data1.qcow2-BACKUPING_NOW" /usr/bin/ssh server1.somedomain.us "/usr/bin/rm /Virtualization/linux/somedomain.com/somedomain.com.ncloud-data2.qcow2-BACKUPING_NOW" /usr/bin/pigz --best --rsyncable somedomain.com.ncloud.qcow2 /usr/bin/pigz --best --rsyncable somedomain.com.ncloud-swap.qcow2 /usr/bin/pigz --best --rsyncable somedomain.com.ncloud-data1.qcow2 /usr/bin/pigz --best --rsyncable somedomain.com.ncloud-data2.qcow2 There is no error on script (commands) execution: [root@server1 ~]# /Backuping/bin/simple_KVM_backup.pl --depth 7 --path Daily --hosting_server server1.somedomain.us ###Backuping VM somedomain.com.ncloud: + Moving somedomain.com.ncloud.xml.06 to somedomain.com.ncloud.xml.07. + Moving somedomain.com.ncloud.qcow2.gz.06 to somedomain.com.ncloud.qcow2.gz.07. ... + Disk snapshot somedomain.com.ncloud-SNAPSHOT for somedomain.com.ncloud ... Domain snapshot somedomain.com.ncloud-SNAPSHOT created Name Creation Time State ----------------------------------------------------------------------- somedomain.com.ncloud-SNAPSHOT 2019-06-18 23:00:04 +0200 disk-snapshot Done! + Time scp start : 2019-06-18 23:00:05 somedomain.com.ncloud.qcow2 100% 20GB 401.3MB/s 00:51 Done! somedomain.com.ncloud-swap.qcow2 100% 4097MB 483.5MB/s 00:08 Done! somedomain.com.ncloud-data1.qcow2 100% 100GB 400.7MB/s 04:15 Done! somedomain.com.ncloud-data2.qcow2 100% 100GB 440.6MB/s 03:52 Done! + Time scp end : 2019-06-18 23:09:14 + Time blockcommit start : 2019-06-18 23:09:14 Successfully pivoted Successfully pivoted Successfully pivoted Successfully pivoted + Time blockcommit end : 2019-06-18 23:09:16 + Snapshot delete ... Domain snapshot somedomain.com.ncloud-SNAPSHOT deleted Done! + Delete snapshot files ... Done! + Time compress start : 2019-06-18 23:09:17 Regards, saso
On Wed, Jun 19, 2019 at 14:23:43 +0200, Saso Tavcar wrote:
Hi,
Recently We've upgraded some KVM hosts from Fedora 29 to Fedora 30 and now experience broken VM configurations on filesystem after virsh blockcommit.
Commands "virsh dumpxml ..." and "virsh dumpxml --inactive ..." is showing diffrent configuration than the one on filesystem. In case of restart libvirtd or system reboot, there are broken VM xml configurations on filesystem.
Hi, this looks like a known problem we dealt with in this thread: https://www.redhat.com/archives/libvir-list/2019-May/msg00416.html It took a few backs-and-forths until I understood it so please read the whole thread.
participants (3)
-
Peter Krempa -
Saso Tavcar -
Saso Tavcar