4:20 p.m.
On 06/03/2017 04:22 PM, Martin Kletzander wrote:
On Sat, Jun 03, 2017 at 09:22:58AM -0400, Michael C Cambria wrote:
>
>
> On 06/02/2017 09:53 AM, Michael C. Cambria wrote:
>>
>>
>> On 06/02/2017 09:43 AM, Martin Kletzander wrote:
>>> [adding back the ML, you probably hit reply instead of reply-all, this
>>> way other people might help if they know more]
>>>
>>> On Fri, Jun 02, 2017 at 08:10:01AM -0400, Michael C. Cambria wrote:
>>>>
>>>> Hi,
>>>>
>>>> libvirtd never seems to get notified that there is work to do.
>>>> journalct
>>>> -f indicated that nothing was logged when connections were attempted
>>>> via
>>>> virsh.
>>>>
>>>> I also tried 'LIBVIRT_DEBUG=1 libvirtd --verbose' and once
startup
>>>> finished, there were no more log entries even though virsh attempts
>>>> were
>>>> made.
>>>>
>>>
>>> That's because it gets overridden by the configuration files. This
>>> might be a bug, but it's not related to what's happening.
>>>
>>>> "ps ax" shows about a dozen "qemu-system-alpha"
processes. I don't
>>>> know
>>>> if it matters but I didn't expect to see this. I didn't
>>>> intentionally
>>>> configure alpha emulations (assuming that's what it is) and
certainly
>>>> don't want to waste resources having it running.
>>>>
>>>
>>> Libvirt caches the capabilities of the emulators it can find in your
>>> system in order not to waste resources. These processes are
>>> expected to
>>> go away after they reply with all libvirt asks them for. However, it
>>> seems like the initialization cannot be completed precisely due to the
>>> fact that these processes don't communicate.
>>>
>>> There might be some details about qemu-system-alpha that are different
>>> when compared to, e.g. qemu-system-x86 and libvirt is not (yet)
>>> adapted
>>> to them, but I installed that emulator and libvirt daemon runs as
>>> usual. It looks like a problem in QEMU. Could you, as a workaround,
>>> try uninstalling that qemu binary from your system and restarting the
>>> service?
>>>
>
> With qemu-system-alpha gone, I now see qemu-system-arm. With all
> qemu-system-* removed (using dnf remove) and stopping, then starting
> libvirtd, I still see qemu-system-arm even though the executable doesn't
> exist:
>
> # ls -al /usr/bin | grep qemu-system
> -rwxr-xr-x. 1 root root 9993416 Apr 13 21:10 qemu-system-i386
> -rwxr-xr-x. 1 root root 10036032 Apr 13 21:10 qemu-system-x86_64
> # systemctl status libvirtd.service
> ● libvirtd.service - Virtualization daemon
> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
> vendor preset: enabled)
> Active: active (running) since Sat 2017-06-03 09:12:19 EDT; 8min ago
> Docs: man:libvirtd(8)
> http://libvirt.org
> Main PID: 5965 (libvirtd)
> Tasks: 23 (limit: 4915)
> Memory: 128.2M
> CPU: 411ms
> CGroup: /system.slice/libvirtd.service
> ├─1537 /usr/bin/qemu-system-arm -S -no-user-config
> -nodefaults -nographic -M none -qmp
> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n
> ├─1638 /usr/bin/qemu-system-arm -S -no-user-config
> -nodefaults -nographic -M none -qmp
> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,n
> ├─5965 /usr/sbin/libvirtd
> ├─5999 /usr/bin/qemu-system-i386 -S -no-user-config
> -nodefaults -nographic -M none -qmp
> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,
> └─6001 /usr/bin/qemu-system-i386 -S -no-user-config
> -nodefaults -nographic -M none -qmp
> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,
>
> Jun 03 09:12:19 example.com systemd[1]: Starting Virtualization
> daemon...
> Jun 03 09:12:19 example.com systemd[1]: Started Virtualization daemon.
>
Those look like leftovers from the previous run. I have no idea why
they keep running, probably the same bug why they are not responding.
I dnf
remove'ed every qemu-system-* in /usr/bin that dnf allowed me to.
The only things are:
$ ls -al /usr/bin | grep qemu-system
-rwxr-xr-x. 1 root root 9993416 Apr 13 21:10 qemu-system-i386
-rwxr-xr-x. 1 root root 10036032 Apr 13 21:10 qemu-system-x86_64
$
Is there a way to remove them? I still have the original problem,
though at least arm and alpha don't show up.
$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
Active: active (running) since Sat 2017-06-03 16:59:40 EDT; 7min ago
Docs: man:libvirtd(8)
http://libvirt.org
Main PID: 1200 (libvirtd)
Tasks: 20 (limit: 4915)
Memory: 101.4M
CPU: 361ms
CGroup: /system.slice/libvirtd.service
├─1200 /usr/sbin/libvirtd
├─1481 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
└─1658 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
Jun 03 16:59:32 eastie.fid4.com systemd[1]: Starting Virtualization
daemon...
Jun 03 16:59:40 eastie.fid4.com systemd[1]: Started Virtualization daemon.
$
Would you mind trying one more thing: Stopping the service, killing
all
these processes, and then starting it? Let's leave the QEMU issue on a
side until we figure this out first, then we can focus on the second
issue. I must say this is quite bizarre.
As requested...
$ sudo systemctl stop libvirtd.service
$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
Active: inactive (dead) since Sat 2017-06-03 17:09:16 EDT; 1s ago
Docs: man:libvirtd(8)
http://libvirt.org
Process: 1200 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
(code=exited, status=0/SUCCESS)
Main PID: 1200 (code=exited, status=0/SUCCESS)
Tasks: 3 (limit: 4915)
Memory: 92.4M
CPU: 386ms
CGroup: /system.slice/libvirtd.service
├─1481 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
└─1658 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
Jun 03 16:59:32 exampl.com systemd[1]: Starting Virtualization daemon...
Jun 03 16:59:40 example.com systemd[1]: Started Virtualization daemon.
Jun 03 17:09:16 example.com systemd[1]: Stopping Virtualization daemon...
Jun 03 17:09:16 example.com systemd[1]: Stopped Virtualization daemon.
$ ps ax | grep virt
1481 ? Sl 0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
1658 ? S 0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
3111 pts/1 S+ 0:00 grep --color=auto virt
$ sudo kill -9 1481
$ sudo kill -9 1658
$ ps ax | grep virt
3120 pts/1 S+ 0:00 grep --color=auto virt
$ sudo systemctl start libvirtd.service
$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
Active: active (running) since Sat 2017-06-03 17:09:52 EDT; 4s ago
Docs: man:libvirtd(8)
http://libvirt.org
Main PID: 3124 (libvirtd)
Tasks: 20 (limit: 4915)
Memory: 103.4M
CPU: 232ms
CGroup: /system.slice/libvirtd.service
├─3124 /usr/sbin/libvirtd
├─3158 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
└─3160 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
Jun 03 17:09:52 example.com systemd[1]: Starting Virtualization daemon...
Jun 03 17:09:52 example.com systemd[1]: Started Virtualization daemon.
$ ps ax | grep virt
3124 ? Ssl 0:00 /usr/sbin/libvirtd
3158 ? Sl 0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
3160 ? S 0:00 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
3166 pts/1 S+ 0:00 grep --color=auto virt
$
Problem still exists.
I also tried stopping libvirtd, renaming both qemu-system-i386 and
qemu-system-x86_64, start libvirtd. Things get further along; dnsmasq
log messages show up.
$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled;
vendor preset: enabled)
Active: active (running) since Sat 2017-06-03 17:14:25 EDT; 7s ago
Docs: man:libvirtd(8)
http://libvirt.org
Main PID: 3246 (libvirtd)
Tasks: 21 (limit: 4915)
Memory: 107.4M
CPU: 300ms
CGroup: /system.slice/libvirtd.service
├─3158 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
├─3160 /usr/bin/qemu-system-i386 -S -no-user-config
-nodefaults -nographic -M none -qmp
unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait
-pidfile /var/lib/libvirt/qemu/capabilities.
├─3246 /usr/sbin/libvirtd
├─3457 /sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/libexec/libvirt_leaseshelper
└─3458 /sbin/dnsmasq
--conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
--dhcp-script=/usr/libexec/libvirt_leaseshelper
Jun 03 17:14:25 example.com libvirtd[3246]: Failed to probe capabilities
for /usr/bin/qemu-kvm: internal error: Child process (LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/bin/qemu-k
Jun 03 17:14:26 example.com dnsmasq[3457]: started, version 2.76
cachesize 150
Jun 03 17:14:26 example.com dnsmasq[3457]: compile time options: IPv6
GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset
auth DNSSEC loop-detect inotify
Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, IP range
192.168.122.2 -- 192.168.122.254, lease time 1h
Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: DHCP, sockets bound
exclusively to interface virbr0
Jun 03 17:14:26 example.com dnsmasq[3457]: reading /etc/resolv.conf
Jun 03 17:14:26 example.com dnsmasq[3457]: using nameserver 172.16.8.1#53
Jun 03 17:14:26 example.com dnsmasq[3457]: read /etc/hosts - 2 addresses
Jun 03 17:14:26 example.com dnsmasq[3457]: read
/var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jun 03 17:14:26 example.com dnsmasq-dhcp[3457]: read
/var/lib/libvirt/dnsmasq/default.hostsfile
$ sudo virsh list
Id Name State
----------------------------------------------------
$
As you can see, none of my vm's show up. I'm guessing the "qemu-kvm:
internal error" results from renaming qemu-system-x86_64??
Could something else (selinux?) be involved?
>
>>> Also, what versions of libvirt and qemu do you have installed?
>>
>> # LIBVIRT_DEBUG=1 libvirtd --verbose
>> 2017-06-02 00:16:30.317+0000: 18088: info : libvirt version: 2.2.1,
>> package: 1.fc25 (Fedora Project, 2017-05-10-22:06:21,
>> buildvm-29.phx2.fedoraproject.org)
>>
>> I'll check on qemu as soon as I can get to the machine. The version
>> should be the latest one gets via 'dnf update' on fedora 25
>
>
> # /usr/sbin/libvirtd --version
> /usr/sbin/libvirtd (libvirt) 2.2.1
> # qemu-x86_64 --version
> qemu-x86_64 version 2.7.1(qemu-2.7.1-6.fc25), Copyright (c) 2003-2016
> Fabrice Bellard and the QEMU Project developers
>
> [ gdb output deleted]