- Users - libvirt List Archives

Slow VM start/revert, when trying to start/revert dozens of VMs in parallel
by Petr Beneš 20 May '22

20 May '22

Hi, my problem can be described simply: libvirt can't handle starting dozens of VMs at the same time. (technically, it can, but it's really slow.) We have an AMD machine with 256 logical cores and 1.5T ram. On that machine there is roughly 200 VMs. Each VM is the same: 8GB of RAM, 4 VCPUs. Half of them is Win7 x86, the other half is Win7 x64. VMs are using qcow2 as the disk image. These images reside in the ramdisk (tmpfs). We use these machines for automatic malware analysis, so our scenario consists of this cycle: - reverting VM to a running state - execute sample inside of the VM for ~1-2 minutes - shutdown the VM Of course, this results in multiple VMs trying to start at the same time. At first, reverts/starts are really fast - second or two. After about a minute, the "revertToSnapshot" suddenly takes 10-15 seconds, which is really unacceptable. For comparison, we're running the same scenarion on Proxmox, where the revertToSnapshot usually takes 2 seconds. Few notes: - Because of this fast cycle (~2-3 minutes) and because of VMs taking 10-15 seconds to start, there is barely more than 25-30 VMs running at once. We would really love to utilise the whole potential of such beast machine of ours, and have at least ~100 VMs running at any given time. - During the time running, the avg. CPU load isn't higher than 25%. Also, there's only about 280 GB of RAM used. Therefore, it's not limitation of our resources. - When the framwork is running and libvirt is making its best to start our VMs, I noticed that every libvirt operation is suddenly very slow. Even simple "virsh list [--all]" takes few seconds to complete, even though it finishes instantly when no VM is running/starting. I was trying to search for this issue, but didn't really find anything besides this presentation: https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Scalability… However, I couldn't find those commits in your upstream. Is this a known issue? Or is there some setting I don't know of which would magically make the VMs start faster? As for steps to reproduce - I don't think there is anything special needed. Just try to start/destroy several VMs in a loop. There is even provided one-liner for that in the presentation above. ``` # For multiple domains: # while virsh start $vm && virsh destroy $vm; do : ; done # → ~30s hang ups of the libvirtd main loop ``` Best Regards, Petr

2 3

macvtap with disconnected physical interface
by Gionatan Danti 19 May '22

19 May '22

Dear list, I just discovered the hard way that if the lower lever physical interface of a macvlan bridge is disconnected (ie: by unplugging the eth cable, resulting in no carrier), inter-guest network traffic from all virtual machines bound to the disconnected interface is dropped. This behavior surprises me, as with the classic bridges I can disconnect the underlying physical interface without causing any harm to inter-guest traffic. Am I doing something wrong, or this really is the expected behavior? If so, can I force the macvtap interfaces to bridge traffic even when the underlying physical interface is disconnected? Regards. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti(a)assyoma.it - info(a)assyoma.it GPG public key ID: FF5F32A8

4 8

Updating domains definitions via API
by Darragh Bailey 17 May '22

17 May '22

Hi, Looking into a bug in vagrant-libvirt where an error during the update will cause the domain to be completely discarded. https://github.com/vagrant-libvirt/vagrant-libvirt/issues/949 Basically I think it stems from doing an undefine -> create with XML new process, which if there is an issue with the new XML due to KVM module not loaded or something similar it will be rejected, but unfortunately it is also unlikely to allow the old definition to be restored either. I'm looking around to try and see if there is an API (specfically in ruby-libvirt) for updating the domain definition, so that if the new XML is rejected at least the old definition remains, and so far I'm drawing a blank. Is the only option here to write using a temporary domain name, then remove the old domain and rename the new definition to the old domain? Or have I missed the obvious API analogous to the edit functionality? -- Darragh Bailey "Nothing is foolproof to a sufficiently talented fool" - unknown

3 13

Problem calling 'virsh' in a script
by Digimer 16 May '22

16 May '22

3 4

Any size limitation on the backing store?
by Jeff Brown 10 May '22

10 May '22

We have a few QCOW2 images that have grown rather large (> 250GB) and to back them up across the VLAN is resource intensive and time consuming. What I'd like to do to optimise backups is freeze the backing store and always run the VMs in 'snapshot 1', and then run 'snapshot 2' to backup 'snapshot 1' to reciprocal failover hypervisors in the DC. I do know it's common practice to even run multiple VMs off a single "golden image" backing store, but I'm concerned about the performance impact with such a large backing store, and 'snapshot 1' growing at ~10GB a year. With that in mind, it should be feasible to pivot 'snapshot 1' into the backing image periodically. Further, I am aware of incremental backups which Debian 11 doesn't yet support. I think it's coming in Debian 12, but some of our servers are actually still on Debian 10. jeff@abacus:/var/lib/libvirt/images$ virsh backup-begin dev error: Operation not supported: incremental backup is not supported yet In a nutshell, is it feasible to run off an > 250GB and growing backing store? Thanks in advance for any advice. -- Jeff Brown Future Foundation Cell 074 101 5170 VoIP 087 550 1210 Fax: 086 532 3508

1 0

virtqemud high CPU load
by lejeczek 06 May '22

06 May '22

Hi guys, Anybody on centos9? With latest updates I get 100% load (so one core fully loaded) almost all the time, I don't remember that being the case before. 'virtqemud' itself does not log much, with default verbosity. Any thoughts? many thanks, L.

1 0

Issue with Guest and 2 Displays
by Thomas Luening 29 Apr '22

29 Apr '22

Hi @ all Since switching from Debian 10 to Debian 11, I've had a new small unsolved problem... with essentially the same customizing as before. It all takes place on my local Desktop-PC, where I have set up a restricted VM for actions in the internet. After I started the VM locally on my PC, I can operate the VM with the virtviewer without any errors. If I then open the second monitor via the Menu 'View->Display', a blank white window appears with the message "Waiting for display 2". xrandr correctly shows the second display as disconnected and also all available display-modes. After the following command xrandr --output Virtual-2 --auto --right-of Virtual-1 both (!) displays are white and both are waiting for a display.... and nothing works anymore. A previously started SSH connection to the VM from the PC shows the following for this moment: tom@pc:~ $ ssh 192.168.100.10 $ tom@vm:~ $su- Password: root@vm:~ # journalctl -f Apr 29 16:24:34 internet kernel: [drm] driver is in bug mode Apr 29 16:24:44 internet spice-vdagent[809]: Unable to find a display id for output index 2) Apr 29 16:24:44 internet spice-vdagent[809]: Unable to find a display id for output index 3) Apr 29 16:24:44 internet spice-vdagent[809]: Unable to find a display id for output index 2) Apr 29 16:24:44 internet spice-vdagent[809]: Unable to find a display id for output index 3) Apr 29 16:24:44 internet kernel: input: spice vdagent tablet as /devices/virtual/input/input7 It is no longer possible to restart the VM via "systemctl reboot", the VM is probably stuck in some systemd-timeouts, only "systemctl reboot -ff" leads to a reboot as a hard intervention. Does anyone know this problem? Is there a solution for this, to get the second display working? Below some relevant system-informations of my PC and the VM. root@pc:~ $ lsb_release -a Distributor ID: Debian Description: Debian GNU/Linux 11 (bullseye) Release: 11 Codename: bullseye root@pc:~ # uname -a Linux pc 5.10.0-13-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux root@pc:~ # cat /etc/libvirt/qemu/internet.xml | grep '<video' -A 3 <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='2' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> tom@vm:~ $ lsb_release -a Distributor ID: Debian Description: Debian GNU/Linux 11 (bullseye) Release: 11 Codename: bullseye tom@vm:~ $ dpkg -l | grep -i spice ii spice-vdagent 0.20.0-2 amd64 Spice agent for Linux tom@vm:~ $ systemctl status spice-vdagent.service ● spice-vdagentd.service - Agent daemon for Spice guests Loaded: loaded (/lib/systemd/system/spice-vdagentd.service; indirect; vendor preset: enabled) Active: active (running) since Fri 2022-04-29 16:00:56 CEST; 16min ago TriggeredBy: ● spice-vdagentd.socket Process: 847 ExecStart=/usr/sbin/spice-vdagentd $SPICE_VDAGENTD_EXTRA_ARGS (code=exited, status=0/SUCCESS) Main PID: 849 (spice-vdagentd) Tasks: 2 (limit: 4702) Memory: 944.0K CPU: 1.918s CGroup: /system.slice/spice-vdagentd.service └─849 /usr/sbin/spice-vdagentd Thanks and Best Regards Tom

1 0

Re: how to change emulator path during live migration
by Peter Krempa 27 Apr '22

27 Apr '22

[re-adding libvirt-users list] Please always reply to the list so that the follow-up conversation is archived and delivered to all subscribers. On Wed, Apr 27, 2022 at 15:36:54 +0800, Jiatong Shen wrote: > Thank you for the feedback! > > Is it ok if the source node does not contain a emulator path used by the > dest node? for example, on src emulator path is /a/b/c, but > on dest it is /a/b/d, and /a/b/d does not exist on src. You can change the emulator path arbitrarily. The only limitation is that the emulator you pick (the binary, not the path) must be able to run the VM, but that will be validated during the migration.

2 1

how to change emulator path during live migration
by Jiatong Shen 27 Apr '22

27 Apr '22

Hello libvirt experts, I am facing the following exceptions during live migrating a virtual machine from one compute node to another. file or directory: libvirt.libvirtError: Cannot check QEMU binary /usr/bin/kvm-spice: No such file or directory File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker rv = meth(*args, **kwargs) File "/var/lib/openstack/lib/python3.6/site-packages/libvirt.py", line 1745, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirt.libvirtError: Cannot check QEMU binary /usr/bin/kvm-spice: No such file or directory After some investigation, we found that this error is triggered because we do not have qemu-kvm installed in our container, btw the libvirt is directly installed on the source node. I have following questions Is it possible to change emulator during live migration? I try to to remove emulator under devices but looks like it does not help. thank you very much and looking forward for the feedback. -- Best Regards, Jiatong Shen

2 1

set fixed time at vm guest startup?
by Fred Clift 26 Apr '22

26 Apr '22

I'm looking at the <clock> libvirt parameters. Is there a way to set the guests clock to a specific time/date at virt startup? I'm trying to virtualize a system with pci passthrough for a hardware device that wants to always live in pre-2011. The license manager for the software/hardware has 31-bit overflow of time calculations. The current process on the physical machine is to power it on, use the bios to set the hardware date to Jan 1 2010, boot the system, use an RC script to ntp-set the date now that the hardware is up and running, and then everything works till next boot. We even have a nice rc script that will set the clock back to 2010 if you do a clean shutdown/reboot. I see I can set my clock sync to variable and specify a negative offset - is there a way to just say "always be jan 10 2010 when you power on"? Fred

3 2