On 06/03/2017 04:22 PM, Martin Kletzander wrote:
On Sat, Jun 03, 2017 at 09:22:58AM -0400, Michael C Cambria wrote:
On 06/02/2017 09:53 AM, Michael C. Cambria wrote:
On 06/02/2017 09:43 AM, Martin Kletzander wrote:
[adding back the ML, you probably hit reply instead of reply-all, this way other people might help if they know more]
On Fri, Jun 02, 2017 at 08:10:01AM -0400, Michael C. Cambria wrote:
Hi,
libvirtd never seems to get notified that there is work to do. journalct -f indicated that nothing was logged when connections were attempted via virsh.
I also tried 'LIBVIRT_DEBUG=1 libvirtd --verbose' and once startup finished, there were no more log entries even though virsh attempts were made.
That's because it gets overridden by the configuration files. This might be a bug, but it's not related to what's happening.
"ps ax" shows about a dozen "qemu-system-alpha" processes. I don't know if it matters but I didn't expect to see this. I didn't intentionally configure alpha emulations (assuming that's what it is) and certainly don't want to waste resources having it running.
Libvirt caches the capabilities of the emulators it can find in your system in order not to waste resources. These processes are expected to go away after they reply with all libvirt asks them for. However, it seems like the initialization cannot be completed precisely due to the fact that these processes don't communicate.
There might be some details about qemu-system-alpha that are different when compared to, e.g. qemu-system-x86 and libvirt is not (yet) adapted to them, but I installed that emulator and libvirt daemon runs as usual. It looks like a problem in QEMU. Could you, as a workaround, try uninstalling that qemu binary from your system and restarting the service?
With qemu-system-alpha gone, I now see qemu-system-arm. With all qemu-system-* removed (using dnf remove) and stopping, then starting libvirtd, I still see qemu-system-arm even though the executable doesn't exist:
# ls -al /usr/bin | grep qemu-system -rwxr-xr-x. 1 root root 9993416 Apr 13 21:10 qemu-system-i386 -rwxr-xr-x. 1 root root 10036032 Apr 13 21:10 qemu-system-x86_64 # systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2017-06-03 09:12:19 EDT; 8min ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 5965 (libvirtd) Tasks: 23 (limit: 4915) Memory: 128.2M CPU: 411ms CGroup: /system.slice/libvirtd.service ├─1537 /usr/bin/qemu-system-arm -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n ├─1638 /usr/bin/qemu-system-arm -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n ├─5965 /usr/sbin/libvirtd ├─5999 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server, └─6001 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,
Jun 03 09:12:19 example.com systemd[1]: Starting Virtualization daemon... Jun 03 09:12:19 example.com systemd[1]: Started Virtualization daemon.
Those look like leftovers from the previous run. I have no idea why they keep running, probably the same bug why they are not responding. I dnf remove'ed every qemu-system-* in /usr/bin that dnf allowed me to. The only things are:
$ ls -al /usr/bin | grep qemu-system -rwxr-xr-x. 1 root root 9993416 Apr 13 21:10 qemu-system-i386 -rwxr-xr-x. 1 root root 10036032 Apr 13 21:10 qemu-system-x86_64 $ Is there a way to remove them? I still have the original problem, though at least arm and alpha don't show up. $ sudo systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2017-06-03 16:59:40 EDT; 7min ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 1200 (libvirtd) Tasks: 20 (limit: 4915) Memory: 101.4M CPU: 361ms CGroup: /system.slice/libvirtd.service ├─1200 /usr/sbin/libvirtd ├─1481 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. └─1658 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. Jun 03 16:59:32 eastie.fid4.com systemd[1]: Starting Virtualization daemon... Jun 03 16:59:40 eastie.fid4.com systemd[1]: Started Virtualization daemon. $
Would you mind trying one more thing: Stopping the service, killing all these processes, and then starting it? Let's leave the QEMU issue on a side until we figure this out first, then we can focus on the second issue. I must say this is quite bizarre.
As requested... $ sudo systemctl stop libvirtd.service $ sudo systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: inactive (dead) since Sat 2017-06-03 17:09:16 EDT; 1s ago Docs: man:libvirtd(8) http://libvirt.org Process: 1200 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS) Main PID: 1200 (code=exited, status=0/SUCCESS) Tasks: 3 (limit: 4915) Memory: 92.4M CPU: 386ms CGroup: /system.slice/libvirtd.service ├─1481 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. └─1658 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. Jun 03 16:59:32 exampl.com systemd[1]: Starting Virtualization daemon... Jun 03 16:59:40 example.com systemd[1]: Started Virtualization daemon. Jun 03 17:09:16 example.com systemd[1]: Stopping Virtualization daemon... Jun 03 17:09:16 example.com systemd[1]: Stopped Virtualization daemon. $ ps ax | grep virt 1481 ? Sl 0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize 1658 ? S 0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize 3111 pts/1 S+ 0:00 grep --color=auto virt $ sudo kill -9 1481 $ sudo kill -9 1658 $ ps ax | grep virt 3120 pts/1 S+ 0:00 grep --color=auto virt $ sudo systemctl start libvirtd.service $ sudo systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2017-06-03 17:09:52 EDT; 4s ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 3124 (libvirtd) Tasks: 20 (limit: 4915) Memory: 103.4M CPU: 232ms CGroup: /system.slice/libvirtd.service ├─3124 /usr/sbin/libvirtd ├─3158 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. └─3160 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. Jun 03 17:09:52 example.com systemd[1]: Starting Virtualization daemon... Jun 03 17:09:52 example.com systemd[1]: Started Virtualization daemon. $ ps ax | grep virt 3124 ? Ssl 0:00 /usr/sbin/libvirtd 3158 ? Sl 0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize 3160 ? S 0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize 3166 pts/1 S+ 0:00 grep --color=auto virt $ Problem still exists. I also tried stopping libvirtd, renaming both qemu-system-i386 and qemu-system-x86_64, start libvirtd. Things get further along; dnsmasq log messages show up. $ sudo systemctl status libvirtd.service ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2017-06-03 17:14:25 EDT; 7s ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 3246 (libvirtd) Tasks: 21 (limit: 4915) Memory: 107.4M CPU: 300ms CGroup: /system.slice/libvirtd.service ├─3158 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. ├─3160 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -M none -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities. ├─3246 /usr/sbin/libvirtd ├─3457 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper └─3458 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper Jun 03 17:14:25 example.com libvirtd[3246]: Failed to probe capabilities for /usr/bin/qemu-kvm: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-k Jun 03 17:14:26 example.com dnsmasq[3457]: started, version 2.76 cachesize 150 Jun 03 17:14:26 example.com dnsmasq[3457]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, sockets bound exclusively to interface virbr0 Jun 03 17:14:26 example.com dnsmasq[3457]: reading /etc/resolv.conf Jun 03 17:14:26 example.com dnsmasq[3457]: using nameserver 172.16.8.1#53 Jun 03 17:14:26 example.com dnsmasq[3457]: read /etc/hosts - 2 addresses Jun 03 17:14:26 example.com dnsmasq[3457]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: read /var/lib/libvirt/dnsmasq/default.hostsfile $ sudo virsh list Id Name State ---------------------------------------------------- $ As you can see, none of my vm's show up. I'm guessing the "qemu-kvm: internal error" results from renaming qemu-system-x86_64?? Could something else (selinux?) be involved?
Also, what versions of libvirt and qemu do you have installed?
# LIBVIRT_DEBUG=1 libvirtd --verbose 2017-06-02 00:16:30.317+0000: 18088: info : libvirt version: 2.2.1, package: 1.fc25 (Fedora Project, 2017-05-10-22:06:21, buildvm-29.phx2.fedoraproject.org)
I'll check on qemu as soon as I can get to the machine. The version should be the latest one gets via 'dnf update' on fedora 25
# /usr/sbin/libvirtd --version /usr/sbin/libvirtd (libvirt) 2.2.1 # qemu-x86_64 --version qemu-x86_64 version 2.7.1(qemu-2.7.1-6.fc25), Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
[ gdb output deleted]