July 2020 - Users - libvirt List Archives

libvirt segfaults with "internal,error: Missing monitor reply object", during block live-migration
by Alex Walender 31 Jul '20

31 Jul '20

Dear libvirt community, Using recent Ubuntu Stein Cloud Packages, we are observing random libvirtd live-migration crashes on the target host. Libvirt is having a SEGFAULT with the qemu driver. Transferring block devices usually works without issues. However, the following memory transfer is causing the target libvirtd randomly to close down its socket, resulting in a roll-backed migration process.I can reproduce this with large VMs, which have a large memory pool. The last error message we see in libvirt logs is: error : qemuMonitorJSONCommandWithFd:315 : internal error: Missing monitor reply object With this, libvirt segfaults and restarts. Before we encountered this issue, we used an older nova-compute package (19.0.3). Not sure if this made a difference with usage of libvirtd-api. After upgrade, we also see a lot of recurring errors during migration: warning : qemuDomainObjBeginJobInternal:7044 : Cannot start job (query, none, none) for domain instance-00008f56; current job is (none, none, migration in) owned by (0 <null>, 0 <null>, 0 remoteDispatchDomainMigratePrepare3Params (flags=0x809b)) for (0s, 0s, 14834s) error : qemuDomainObjBeginJobInternal:7066 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params) They don't abort the running migration process, but spam every minute to the systemd journal. Source and destination run the same packages: Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-99-generic x86_64) OpenStack Stein (Ubuntu Cloud Archive) Libvirt+QEMU_x86 keystone-common 2:15.0.1-0ubuntu1~cloud0 libvirt-daemon 5.0.0-1ubuntu2.6~cloud0 qemu-system-x86 1:3.1+dfsg-2ubuntu3.7~cloud0 neutron-linuxbridge-agent 2:14.2.0-0ubuntu1~cloud0 neutron-plugin-ml2 2:14.2.0-0ubuntu1~cloud0 nova-compute 2:19.2.0-0ubuntu1~cloud0 nova-compute-libvirt 2:19.2.0-0ubuntu1~cloud0 I have attached source/destination debug logs from libvirtd and nova-compute here: https://denzelx.ddns.net/index.php/s/KPJ7vv4aTcb69XD Any help would be nice! Best Regards -- M.Sc Alex Walender de.NBI Cloud Bielefeld Administrator Center for Biotechnology (CeBiTec) University of Bielefeld 33594 Bielefeld Germany room: M3-118 phone: +49 (521) 106 2907

3 3

Routed network can't reach outside network
by Rui Correia 28 Jul '20

28 Jul '20

Greetings folks. I've setup libvirtd on my manjaro linux laptop. Got a couple of VM's running (Win10 and Debian10) through NAT without any issues. This is what the current network diagram looks like and it works fine: +-----------------------------------+ | +---------------------+ | | | +----------+ | | | | |Win 10 VM | | | | | |10.1.1.10 | | | | | +----------+ | | | Laptop | | | | Manjaro | +-------------+ | | | 10.0.0.10 | |Debian 10 VM | | | +-------->+ | |10.1.1.11 | | | | | | +-------------+ | | | | |NAT | | | | |10.1.1.0/24 | | | | +---------------------+ | +------------+ | +-----------------------------------+ |router | | |switch +---+ |10.0.0.0/24 | | +---------+ +------------+ | |Desktop | +-------->+Manjaro | |10.0.0.11| +---------+ But now I need the debian machine to be accessible from another host on the lan 10.0.0.0/24 which of course is outside the host. That network diagram would look like this: +-----------------------------------------+ | +------------------+ | | | +----------+ | | | | |Win 10 VM | | | | | |10.1.1.10 | | | | | NAT +----------+ | | | | 10.1.1.0/24 | | | +------------------+ | | Laptop | +-------->+ Manjaro +------------------------+ | | | 10.0.0.10 | +-------------+ | | | | | |Debian 10 VM | | | | | | |10.2.2.10 | | | | | | Routed +-------------+ | | +------------+ | | | 10.2.2.0/24 | | |router | | | +------------------------+ | |switch +---+ +-----------------------------------------+ |10.0.0.0/24 | | +------------+ | | | +---------+ | |Desktop | +-------->+Manjaro | |10.0.0.11| +---------+ So, I've setup a 'routed network' for the Debian 10 VM but it's not working as I would expect. The host can ping the Debian VM and the Debian VM can ping the host but the Debian VM cannot ping the router 10.0.0.1 or any ip address on the internet. I've been using Virtual Machine Manager to set everything up. And this is how the routed network is configured <network connections="1"> <name>routed</name> <uuid>970a25f7-29b6-4a6b-b890-f593eae4fc15</uuid> <forward dev="wlo1" mode="route"> <interface dev="wlo1"/> </forward> <bridge name="virbr2" stp="on" delay="0"/> <mac address="52:54:00:bf:35:42"/> <domain name="routed"/> <ip address="10.2.2.1" netmask="255.255.255.0"> <dhcp> <range start="10.2.2.2" end="10.2.2.254"/> </dhcp> </ip> </network> Any idea on what i might be doing wrong? Thanks in advance. Cheers Rui Correia

6 9

machinectl does not show libvirtd vms
by Ismael Bouya 27 Jul '20

27 Jul '20

(this is a follow-up of a discussion on IRC with mprivozn) Hello, I’m facing an issue where systemd-machined is not aware of new VMs happening in libvirt (with QEMU). I attached debug logs for a VM starting. To repeat the discussion that happened on IRC: - libvirt config does not specify anything related to cgroup_controllers - cgroups are mounted (mount | grep cgroup is not empty) - in libvirtd.service I see no mention of cgroups (apart from comments related to TasksMax and LimitMEMLOCK) I’m suspecting a packaging issue in my distribution (NixOS) but I’m not really sure where I should look at to solve it. NB: the reason I wanted machinectl was to make use of nsswitch mymachines facilities, which I learnt wouldn’t work anyway. So I’m only reporting it in case something needs to be fixed (probably in my distribution packaging). -- Ismael

1 0

Re: Routed network can't reach outside network
by Rui Correia 23 Jul '20

23 Jul '20

Hi Ken, Of course I did it all by hand. :-D No, I didn't. I'm a fraud. I used http://asciiflow.com/ which asks you to register in order to save your drawings in their cloud. If you don't register, you can still draw and export your drawings, but you can't save them for reusing those later. I see I've missed a couple of posts in this small thread. I was under the impression that I had subscribed to the list but apparently I'm only receiving the messages directly addressed to me. Hence why I took so much time to reply. I'll try to subscribe again. Thank you all in advance. Cheers, Rui Correia On Thu, Jul 23, 2020 at 3:31 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > > you must be an ASCII charting demigod. Did you use software to make > those, or do > > them yourselves? Either way, I'm impressed... > > Search for AsciiArtStudio.exe > > > > >

1 0

virt-install missing parameters
by David 22 Jul '20

22 Jul '20

Hello, Hopefully this is the right place to send a question like this... I'm attempting to automate creation of VMs using virt-install and a cloud-init disk image. To get this to work, I need to specify the location of the cloud-init configs by passing smbios key/value pairs. Normally, -smbios is provided to qemu to do this. With virt-install, the Internet says I should use --sysinfo. However, --sysinfo (as well as --qemu-comandline) is reported as an invalid command flag, and it is not present when I look at man virt-install and virt-install --help. I am hoping someone on the list knows why this isn't working or can explain how I might provide the arguments to qemu. The host is Debian 9. It is up-to-date as far as package versions are concerned, as much as Debian 9 can be. I can't find any indication about why the arguments would have been taken away. Thanks!

1 0

host and vm on isolated network, there is ip (via dhcp) but not ping
by daggs 22 Jul '20

22 Jul '20

Greetings, I've setup an vm with openwrt in it, defined a isolated lan between the vm and the host and booted the vm up. I see the vm is up, made sure the vnic is visible in both the host and guest and added it to the br in the guest. I've issued an dhcpd call on the vnic (labeled vnic0) in the host and got an ip, see: dagg@NCC-5001D ~ $ dhcpcd vnet0 DUID 00:01:00:01:23:dd:d8:5b:e0:d5:5e:d9:f2:e2 vnet0: IAID 00:10:20:bf vnet0: rebinding lease of 192.168.1.130 vnet0: probing address 192.168.1.130/24 vnet0: soliciting an IPv6 router vnet0: leased 192.168.1.130 for 43200 seconds vnet0: adding route to 192.168.1.0/24 vnet0: adding default route via 192.168.1.1 forked to background, child pid 26279 dagg@NCC-5001D ~ $ ifconfig virtsw0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 52:54:00:3e:3f:88 txqueuelen 1000 (Ethernet) RX packets 123098 bytes 16327962 (15.5 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 6 bytes 252 (252.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.130 netmask 255.255.255.0 broadcast 192.168.1.255 inet6 fe80::fc54:ff:fe10:20bf prefixlen 64 scopeid 0x20<link> ether fe:54:00:10:20:bf txqueuelen 1000 (Ethernet) RX packets 45 bytes 8002 (7.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 39 bytes 2676 (2.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 dagg@NCC-5001D ~ $ ping 192.168.1.1 PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. ^C --- 192.168.1.1 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1018ms the vm's xml can be found at https://pastebin.com/1gXBGcPb virtsw0 is defined as follows: <network connections='1'> <name>virtsw0</name> <uuid>c8eb15a3-cc5c-4bd6-8f3b-5790792ddccc</uuid> <bridge name='virtsw0' stp='on' delay='0'/> <mac address='52:54:00:3e:3f:88'/> </network> the os is gentoo, the versions are libvirt-6.2.0 qemu-5.0.0. I have another server running debian 10 with the same virtsw0 definition, there the connection is working. /var/lib/libvirt/dnsmasq/virtsw0.macs has only [] in it, can that be the issue? Thanks, Dagg.

2 2

dns test 2
by NO SPAM 21 Jul '20

21 Jul '20

please ignore

1 0

incomplete features for external snapshot
by Henry lol 21 Jul '20

21 Jul '20

as far as I know, deleting/reverting features for external snapshot are not being supported for long time. I'm wondering why it takes so long time and what is the main problem even though those tasks can be done manually. thanks.

1 0

dns test
by NO SPAM 20 Jul '20

20 Jul '20

please ignore

1 0

9p file sharing issue
by Johan 19 Jul '20

19 Jul '20

I've run into an issue with 9p file sharing that I suspect is a bug. I'm running some Debian ARM VMs (armhf Ubuntu guests on an aarch64 Debian stretch host in case that matters). I'm using direct kernel boot, so I'm also using 9p host file sharing so that kernel updates from within the guest would work. However, I've noticed that dpkg (the Debian/Ubuntu package manager) fails when writing into a /boot directory that's shared from the host via 9p. Some investigation reveals that this is because dpkg does an openat() call of the type openat(AT_FDCWD, "/boot/vmlinuz-4.15.0-111-generic-lpae.dpkg-new", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 000); and that it's the 0 permissions mode (last parameter) that causes the call to fail with a "Permission denied" error. A simple test program that does the above openat() call but with another mode (say, 600) works just fine. Some further data: * QEMU runs as the unprivileged user libvirt-qemu (Debian default) * QEMU is version 2.8, libvirt version 3.0.0 * XML snippet for 9p setup: <filesystem type='mount' accessmode='squash'> <source dir='/host/path/boot-test9p'/> <target dir='boot_mount'/> <alias name='fs0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> </filesystem> * The 9p filesystem is mounted by this line in /etc/fstab on the guest: boot_mount /boot 9p rw,dirsync,_netdev,relatime,trans=virtio,version=9p2000.L 0 0 The essential problem seems to be that opening a file with a mode of 0 is not compatible with the "squash" 9p access mode. Is this expected? Thank you, Johan

1 0