[libvirt-users] Could not destroy domain, current job is remoteDispatchConnectGetAllDomainStats
by Serhii Kharchenko
Hello libvirt-users list,
We're catching the same bug since 3.4.0 version (3.3.0 works OK).
So, we have process that is permanently connected to libvirtd via socket
and it is collecting stats, listening to events and control the VPSes.
When we try to 'shutdown' a number of VPSes we often catch the bug. One of
VPSes sticks in 'in shutdown' state, no related 'qemu' process is present,
and there is the next error in the log:
Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.005+0000:
20438: warning : qemuGetProcessInfo:1460 : cannot parse process status data
Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000:
20441: error : virFileReadAll:1420 : Failed to open file
'/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d36\x2dDOMAIN1.scope/cpuacct.usage':
No such file or directory
Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000:
20441: error : virCgroupGetValueStr:844 : Unable to read from
'/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d36\x2dDOMAIN1.scope/cpuacct.usage':
No such file or directory
Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000:
20441: error : virCgroupGetDomainTotalCpuStats:3319 : unable to get cpu
account: Operation not permitted
Jan 17 13:54:23 server1 libvirtd[20437]: 2018-01-17 13:54:23.805+0000:
20522: warning : qemuDomainObjBeginJobInternal:4862 : Cannot start job
(destroy, none) for domain DOMAIN1; current job is (query, none) owned by
(20440 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (30s, 0s)
Jan 17 13:54:23 server1 libvirtd[20437]: 2018-01-17 13:54:23.805+0000:
20522: error : qemuDomainObjBeginJobInternal:4874 : Timed out during
operation: cannot acquire state change lock (held by
remoteDispatchConnectGetAllDomainStats)
I think only the last line matters.
The bug is highly reproducible. We can easily catch it even when we call
multiple 'virsh shutdown' in shell one by one.
When we shutdown the process connected to the socket - everything become OK
and the bug is gone.
The system is used is Gentoo Linux, tried all modern versions of libvirt
(3.4.0, 3.7.0, 3.8.0, 3.9.0, 3.10.0, 4.0.0-rc2 (today's version from git))
and they have this bug. 3.3.0 works OK.
Thanks for any help in advance.
We can send any additional info if needed.
~Serhii
6 years, 9 months
[libvirt-users] How to use /dev/net/tun from libvirt-lxc with user namespacing enabled
by Thiago Padilha
I have a container rootfs that I use to keep all work-related stuff. This
container was originally created by lxd (which creates all containers for
use with user namespacing), but now I decided to start using libvirt for
container management since I already use it for virtual machines, which
will spare me from dealing with multiple hypervisor technologies.
I managed to create a working domain xml for the container, and everything
seems to be working very well except one thing: I cannot start openconnect
(VPN software) inside the container. I noticed that by default libvirt
won't create /dev/net/tun for the container, so I added this to the domain
xml:
<filesystem type='mount'>
<source dir='/dev/net'/>
<target dir='/dev/net'/>
</filesystem>
This successfully created /dev/net/tun in the container, but openconnect
still can't open it even though it has 666 permissions. It seems this is
exactly what lxd does to allow VPNs for their unprivileged containers, as
shown by the output of ls -l /dev/net
total 0
crw-rw-rw- 1 nobody nogroup 10, 200 Jan 29 13:23 tun
The same container can also be successfully booted with systemd-nspawn,
also allowing openconnect to create its VPN (though systemd-nspawn appears
to create a new device node, owned by root relative to the user namespace).
I already tried setting security driver to "none" in /etc/libvirt/lxc.conf,
but it had no effect. I get "Operation not permitted" when trying to open
/dev/net/tun, which is also the message openconnect displays in its logs.
Can someone guide me on how I might debug what is causing this error? BTW,
here's the full xml:
<domain type='lxc'>
<name>work-stuff</name>
<uuid>ffee008c-ec6b-48ab-af6d-4aba830847a1</uuid>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<vcpu placement='static'>16</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64'>exe</type>
<init>/sbin/init</init>
</os>
<idmap>
<uid start='0' target='165536' count='65536'/>
<gid start='0' target='165536' count='65536'/>
</idmap>
<cpu mode='host-model'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/var/lib/libvirt/containers/work-stuff/rootfs'/>
<target dir='/'/>
</filesystem>
<filesystem type='mount'>
<source dir='/dev/net'/>
<target dir='/dev/net'/>
</filesystem>
<interface type='network'>
<mac address='52:54:00:3e:59:e9'/>
<source network='default'/>
</interface>
<console type='pty'>
<target type='lxc' port='0'/>
</console>
</devices>
</domain>
6 years, 9 months
[libvirt-users] Cannot add metadata to network XML
by Shashwat shagun
Hi, i tried adding metadata to Network XML ( NOT DOMAIN XML) but it removes
it as soon as i save
here's my network XML
<network>
<name>bridgeTest0</name>
<uuid>cf2aae7a-b459-459b-ba2c-399190607629</uuid>
<metadata>
<app1:foo xmlns:app1="http://app1.org/app1/">
<app1:test>testing!</app1:test>
</app1:foo>
</metadata>
<forward mode='bridge'/>
<bridge name='lxcbr0'/>
</network>
--
Regards,
Shashwat Shagun
6 years, 9 months
[libvirt-users] Connection to libvirt
by Ori Liel
My attempt to install a host using oVirt results in the following message
in the host:
libvirt-buests.sh[3679]: Unable to connect to libvirt currently. Retrying
.. 10Please enter your authentication name: Please enter your password:
after several such retries:
libvirt-buests.sh[3679]: Unable to connect to libvirt currently. Retrying
.. 1Can't connect to deafult. Skipping.
What may be the cause of this?
Thanks,
Ori.
6 years, 9 months
Re: [libvirt-users] issue with openssh-server running in a libvirt based centos virtual machine
by Peter Crowther
You say you can ping but not ssh. If you install tcpdump on the VM, can you
see the ping packets arriving and leaving? If not, I suspect an address
collision - especially if ping continues to work with the VM shut down. If
you can't ping, check the other end of your bridge. I'm more familiar with
open vSwitch, but I'm somewhat concerned that your bridge definition
doesn't include a physical NIC as one of its connections.
Peter
On 27 Jan 2018 1:13 p.m., "Adrian Pascalau" <adrian27oradea(a)gmail.com>
wrote:
Hi,
I have a strange issue in a libvirt environment, and I do not know how
to solve it.
I have two centos hosts: first one is a physical server called
server1, that acts as a host for the second one, called centos1. The
centos1 is a virtual machine (VM) running in server1. A linux bridge
in forwarding mode is used to connect the centos1 VM network interface
to the server1 network interface and to the external network. The
centos1 VM and the linux bridge are managed with libvirt (well, the
bridge itself in this case is created manually).
# virsh net-dumpxml br0
<network connections='1'>
<name>br0</name>
<uuid>5aaf72a5-023d-4b84-9d7c-d68b0918f620</uuid>
<forward mode='bridge'/>
<bridge name='br0'/>
</network>
# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.fc15b4137688 no eno1
vnet0
Both server1 and centos1 have IP addresses in the same subnet, and
both are reachable with ping from every other host in my network. In
both server1 and centos1, the openssh-server configuration in
/etc/ssh/sshd_config is the default one, and has not been changed.
When I ssh with Putty to the physical server server1 IP address,
everything works as expected: I get a login prompt, I enter my
password and I log in.
However, when I use Putty to connect to the centos1 VM, I do not get a
login prompt whatsoever. So I think there might be some issue in
between the server1 physical interface and my centos VM.
I used openssh-server in debug mode, so see where the ssh connection
hangs, and here is what I get:
[...]
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: private host key #0: ssh-rsa
SHA256:pEuFQsodwK+0PoRzbVRba1ahHLEpwp8DG2KGQmxOGJk
debug1: private host key #1: ecdsa-sha2-nistp256
SHA256:F6HrSNWZhYaU7LMweI+RBviqTCHcTYyMBGPDz5OjT4c
debug1: private host key #2: ssh-ed25519
SHA256:aG3V6jjPHXUnNeavbxT/xozqrb5q3yWDkkAmXBCdnGk
debug1: inetd sockets after dupping: 3, 3
Connection from x.x.x.181 port 49436 on x.x.x.115 port 22
debug1: Client protocol version 2.0; client software version
PuTTY_Release_0.70
debug1: no match: PuTTY_Release_0.70
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Enabling compatibility mode for protocol 2.0
debug1: SELinux support enabled [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug1: list_hostkey_types:
ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
[preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
I tried with other windows based ssh clients (MobaXterm) and the same
issue happens. I discussed this with people in the openssh mailing
list, and they said this issue could most probably be caused by a path
MTU/fragmentation problem...
Then I moved my centos1 qcow2 image in another physical server called
server2, with exactly the same hw specs and network connections, where
I have installed an all-in-one OpenStack Pike. The network would be
managed with neutron in this case, however I have configured neutron
exactly so that the centos1 VM interface connects through a linux
bridge (managed by neutron) to the server1 physical network interface,
like in the libvirt case.
# brctl show
bridge name bridge id STP enabled interfaces
brqa13eec69-a4 8000.0e7faabad6d4 no eno1
tap8cb53db0-fb
tapb24a1cc5-20
Above the tapb24a1cc5-20 is the tap interface towards my centos1 VM.
In this case, the Putty issue is gone, and I do not have any issue
anymore. If I go back to the libvirt environment in server1, I get the
same issue again.
So I tend to think that my ssh connection issue is caused by the
libvirt and the way networking is configured, however I do not know
how to troubleshoot this further anymore.
Any help is greatly appreciated.
Adrian
_______________________________________________
libvirt-users mailing list
libvirt-users(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users
6 years, 9 months
[libvirt-users] Static DHCP assignments to VMs connected to bridge
by Dagmawi Biru
Hello libvirt-users,
Given the following setup:
---
[Rest of network/dhcp server] ------[ physical interface] ------ [ bridge]
<----> | <---> [ VM ]
I have an interface configured in my Domain XML like so:
---
<interface type="bridge">
<source bridge="br20">
</interface>
Bridge "br20" is connected to an external network that has a DHCP server.
However, I want to set it so the interface for this VM has a static IP
served to it, either by a locally running DHCP server or via some sort of
static IP assignment in the XML configuration. Is this possible?
6 years, 9 months
[libvirt-users] issue with openssh-server running in a libvirt based centos virtual machine
by Adrian Pascalau
Hi,
I have a strange issue in a libvirt environment, and I do not know how
to solve it.
I have two centos hosts: first one is a physical server called
server1, that acts as a host for the second one, called centos1. The
centos1 is a virtual machine (VM) running in server1. A linux bridge
in forwarding mode is used to connect the centos1 VM network interface
to the server1 network interface and to the external network. The
centos1 VM and the linux bridge are managed with libvirt (well, the
bridge itself in this case is created manually).
# virsh net-dumpxml br0
<network connections='1'>
<name>br0</name>
<uuid>5aaf72a5-023d-4b84-9d7c-d68b0918f620</uuid>
<forward mode='bridge'/>
<bridge name='br0'/>
</network>
# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.fc15b4137688 no eno1
vnet0
Both server1 and centos1 have IP addresses in the same subnet, and
both are reachable with ping from every other host in my network. In
both server1 and centos1, the openssh-server configuration in
/etc/ssh/sshd_config is the default one, and has not been changed.
When I ssh with Putty to the physical server server1 IP address,
everything works as expected: I get a login prompt, I enter my
password and I log in.
However, when I use Putty to connect to the centos1 VM, I do not get a
login prompt whatsoever. So I think there might be some issue in
between the server1 physical interface and my centos VM.
I used openssh-server in debug mode, so see where the ssh connection
hangs, and here is what I get:
[...]
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: private host key #0: ssh-rsa
SHA256:pEuFQsodwK+0PoRzbVRba1ahHLEpwp8DG2KGQmxOGJk
debug1: private host key #1: ecdsa-sha2-nistp256
SHA256:F6HrSNWZhYaU7LMweI+RBviqTCHcTYyMBGPDz5OjT4c
debug1: private host key #2: ssh-ed25519
SHA256:aG3V6jjPHXUnNeavbxT/xozqrb5q3yWDkkAmXBCdnGk
debug1: inetd sockets after dupping: 3, 3
Connection from x.x.x.181 port 49436 on x.x.x.115 port 22
debug1: Client protocol version 2.0; client software version PuTTY_Release_0.70
debug1: no match: PuTTY_Release_0.70
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Enabling compatibility mode for protocol 2.0
debug1: SELinux support enabled [preauth]
debug1: permanently_set_uid: 74/74 [preauth]
debug1: list_hostkey_types:
ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
[preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
I tried with other windows based ssh clients (MobaXterm) and the same
issue happens. I discussed this with people in the openssh mailing
list, and they said this issue could most probably be caused by a path
MTU/fragmentation problem...
Then I moved my centos1 qcow2 image in another physical server called
server2, with exactly the same hw specs and network connections, where
I have installed an all-in-one OpenStack Pike. The network would be
managed with neutron in this case, however I have configured neutron
exactly so that the centos1 VM interface connects through a linux
bridge (managed by neutron) to the server1 physical network interface,
like in the libvirt case.
# brctl show
bridge name bridge id STP enabled interfaces
brqa13eec69-a4 8000.0e7faabad6d4 no eno1
tap8cb53db0-fb
tapb24a1cc5-20
Above the tapb24a1cc5-20 is the tap interface towards my centos1 VM.
In this case, the Putty issue is gone, and I do not have any issue
anymore. If I go back to the libvirt environment in server1, I get the
same issue again.
So I tend to think that my ssh connection issue is caused by the
libvirt and the way networking is configured, however I do not know
how to troubleshoot this further anymore.
Any help is greatly appreciated.
Adrian
6 years, 10 months
[libvirt-users] Create virtual machine failed using virDomainCreateXML
by netsurfed
Hi all,
When I created a virtual machine using virDomainCreateXML, with virtualport type was openvswitch, and virtual machine creation failed. The error message is:
internal error: Child process (/sbin/tc filter add dev vnet110 parent ffff: protocol all u32 match u32 0 0 police rate 0kbps burst 0kb mtu 64kb drop flowid :1) unexpected exit status 2: RTNETLINK answers: Invalid argument
The Domain XML file the <interface> section like this:
<interface type='bridge'><mac address='52:54:00:71:b1:b6'/><source bridge='ovsbr'/><virtualport type='openvswitch'/><address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/></interface>
I looked at the system log and it looked like an ovs port problem:
4560 Jan 22 17:43:33 ubuntu-24 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exists del-port vnet110 -- add-port lcs_br_1 vnet110 -- set Interface vnet110 "external- ids:attached-mac=\"52:54:00:21:63:25\"" -- set Interface vnet110 "external-ids:iface-id=\"d0e5c770-6f03-4703-b107-3b074f989b06\"" -- set Interface vnet110 "external-ids:vm-id=\"1c79066e- 46a2-4fa0-984c-a88a7fffcd2b\"" -- set Interface vnet110 external-ids:iface-status=active
4561 Jan 22 17:43:33 ubuntu-24 kernel: [29028.418651] device vnet110 entered promiscuous mode
4562 Jan 22 17:43:33 ubuntu-24 libvirtd[4307]: 2018-01-22 09:43:33.507+0000: 4310: error : virCommandWait:2572 : internal error: Child process (/sbin/tc filter add dev vnet110 parent ffff: pr otocol all u32 match u32 0 0 police rate 0kbps burst 0kb mtu 64kb drop flowid :1) unexpected exit status 2: RTNETLINK answers: Invalid argument
4563 Jan 22 17:43:33 ubuntu-24 libvirtd[4307]: We have an error talking to the kernel
4564 Jan 22 17:43:33 ubuntu-24 kernel: [29028.454028] device vnet110 left promiscuous mode
4565 Jan 22 17:43:33 ubuntu-24 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exists del-port vnet110
However, when I use "tc" to manage "vnet*" of an existing virtual machine, it still returns the same error.
root@ubuntu-24:~# /sbin/tc filter add dev vnet83 parent ffff: protocol all u32 match u32 0 0 police rate 0kbps burst 0kb mtu 64kb drop flowid :1
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
Is there any reason for this problem? Thank you very much.
Below some information about my machine:
libvirt version: 3.4.0
ovs_version: "2.8.90"
root@ubuntu-24:~# uname -a
Linux ubuntu-24 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
6 years, 10 months
[libvirt-users] Create virtual machine failed using virDomainCreateXML
by netsurfed
Hi all,
When I created a virtual machine using virDomainCreateXML, with virtualport type was openvswitch, and virtual machine creation failed. The error message is:
internal error: Child process (/sbin/tc filter add dev vnet110 parent ffff: protocol all u32 match u32 0 0 police rate 0kbps burst 0kb mtu 64kb drop flowid :1) unexpected exit status 2: RTNETLINK answers: Invalid argument
The Domain XML file the <interface> section like this:
I looked at the system log and it looked like an ovs port problem.
However, when I use "tc" to manage "vnet*" of an existing virtual machine, it still returns the same error.
root@ubuntu-24:~# /sbin/tc filter add dev vnet83 parent ffff: protocol all u32 match u32 0 0 police rate 0kbps burst 0kb mtu 64kb drop flowid :1
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
Is there any reason for this problem? Thank you very much.
Below some information about my machine:
libvirt version: 3.4.0
ovs_version: "2.8.90"
root@ubuntu-24:~# uname -a
Linux ubuntu-24 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
6 years, 10 months
