January 2018 - Users - libvirt List Archives

[libvirt-users] Could not destroy domain, current job is remoteDispatchConnectGetAllDomainStats
by Serhii Kharchenko 01 Feb '18

01 Feb '18

Hello libvirt-users list, We're catching the same bug since 3.4.0 version (3.3.0 works OK). So, we have process that is permanently connected to libvirtd via socket and it is collecting stats, listening to events and control the VPSes. When we try to 'shutdown' a number of VPSes we often catch the bug. One of VPSes sticks in 'in shutdown' state, no related 'qemu' process is present, and there is the next error in the log: Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.005+0000: 20438: warning : qemuGetProcessInfo:1460 : cannot parse process status data Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000: 20441: error : virFileReadAll:1420 : Failed to open file '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d36\x2dDOMAIN1.scope/cpuacct.usage': No such file or directory Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000: 20441: error : virCgroupGetValueStr:844 : Unable to read from '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d36\x2dDOMAIN1.scope/cpuacct.usage': No such file or directory Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000: 20441: error : virCgroupGetDomainTotalCpuStats:3319 : unable to get cpu account: Operation not permitted Jan 17 13:54:23 server1 libvirtd[20437]: 2018-01-17 13:54:23.805+0000: 20522: warning : qemuDomainObjBeginJobInternal:4862 : Cannot start job (destroy, none) for domain DOMAIN1; current job is (query, none) owned by (20440 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (30s, 0s) Jan 17 13:54:23 server1 libvirtd[20437]: 2018-01-17 13:54:23.805+0000: 20522: error : qemuDomainObjBeginJobInternal:4874 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats) I think only the last line matters. The bug is highly reproducible. We can easily catch it even when we call multiple 'virsh shutdown' in shell one by one. When we shutdown the process connected to the socket - everything become OK and the bug is gone. The system is used is Gentoo Linux, tried all modern versions of libvirt (3.4.0, 3.7.0, 3.8.0, 3.9.0, 3.10.0, 4.0.0-rc2 (today's version from git)) and they have this bug. 3.3.0 works OK. Thanks for any help in advance. We can send any additional info if needed. ~Serhii

5 8

[libvirt-users] How to use /dev/net/tun from libvirt-lxc with user namespacing enabled
by Thiago Padilha 31 Jan '18

31 Jan '18

I have a container rootfs that I use to keep all work-related stuff. This container was originally created by lxd (which creates all containers for use with user namespacing), but now I decided to start using libvirt for container management since I already use it for virtual machines, which will spare me from dealing with multiple hypervisor technologies. I managed to create a working domain xml for the container, and everything seems to be working very well except one thing: I cannot start openconnect (VPN software) inside the container. I noticed that by default libvirt won't create /dev/net/tun for the container, so I added this to the domain xml: <filesystem type='mount'> <source dir='/dev/net'/> <target dir='/dev/net'/> </filesystem> This successfully created /dev/net/tun in the container, but openconnect still can't open it even though it has 666 permissions. It seems this is exactly what lxd does to allow VPNs for their unprivileged containers, as shown by the output of ls -l /dev/net total 0 crw-rw-rw- 1 nobody nogroup 10, 200 Jan 29 13:23 tun The same container can also be successfully booted with systemd-nspawn, also allowing openconnect to create its VPN (though systemd-nspawn appears to create a new device node, owned by root relative to the user namespace). I already tried setting security driver to "none" in /etc/libvirt/lxc.conf, but it had no effect. I get "Operation not permitted" when trying to open /dev/net/tun, which is also the message openconnect displays in its logs. Can someone guide me on how I might debug what is causing this error? BTW, here's the full xml: <domain type='lxc'> <name>work-stuff</name> <uuid>ffee008c-ec6b-48ab-af6d-4aba830847a1</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <vcpu placement='static'>16</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64'>exe</type> <init>/sbin/init</init> </os> <idmap> <uid start='0' target='165536' count='65536'/> <gid start='0' target='165536' count='65536'/> </idmap> <cpu mode='host-model'> <model fallback='allow'/> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/var/lib/libvirt/containers/work-stuff/rootfs'/> <target dir='/'/> </filesystem> <filesystem type='mount'> <source dir='/dev/net'/> <target dir='/dev/net'/> </filesystem> <interface type='network'> <mac address='52:54:00:3e:59:e9'/> <source network='default'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> </devices> </domain>

2 2

[libvirt-users] Cannot add metadata to network XML
by Shashwat shagun 31 Jan '18

31 Jan '18

Hi, i tried adding metadata to Network XML ( NOT DOMAIN XML) but it removes it as soon as i save here's my network XML <network> <name>bridgeTest0</name> <uuid>cf2aae7a-b459-459b-ba2c-399190607629</uuid> <metadata> <app1:foo xmlns:app1="http://app1.org/app1/"> <app1:test>testing!</app1:test> </app1:foo> </metadata> <forward mode='bridge'/> <bridge name='lxcbr0'/> </network> -- Regards, Shashwat Shagun

2 2

[libvirt-users] Connection to libvirt
by Ori Liel 30 Jan '18

30 Jan '18

My attempt to install a host using oVirt results in the following message in the host: libvirt-buests.sh[3679]: Unable to connect to libvirt currently. Retrying .. 10Please enter your authentication name: Please enter your password: after several such retries: libvirt-buests.sh[3679]: Unable to connect to libvirt currently. Retrying .. 1Can't connect to deafult. Skipping. What may be the cause of this? Thanks, Ori.

2 1

Re: [libvirt-users] issue with openssh-server running in a libvirt based centos virtual machine
by Peter Crowther 30 Jan '18

30 Jan '18

You say you can ping but not ssh. If you install tcpdump on the VM, can you see the ping packets arriving and leaving? If not, I suspect an address collision - especially if ping continues to work with the VM shut down. If you can't ping, check the other end of your bridge. I'm more familiar with open vSwitch, but I'm somewhat concerned that your bridge definition doesn't include a physical NIC as one of its connections. Peter On 27 Jan 2018 1:13 p.m., "Adrian Pascalau" <adrian27oradea(a)gmail.com> wrote: Hi, I have a strange issue in a libvirt environment, and I do not know how to solve it. I have two centos hosts: first one is a physical server called server1, that acts as a host for the second one, called centos1. The centos1 is a virtual machine (VM) running in server1. A linux bridge in forwarding mode is used to connect the centos1 VM network interface to the server1 network interface and to the external network. The centos1 VM and the linux bridge are managed with libvirt (well, the bridge itself in this case is created manually). # virsh net-dumpxml br0 <network connections='1'> <name>br0</name> <uuid>5aaf72a5-023d-4b84-9d7c-d68b0918f620</uuid> <forward mode='bridge'/> <bridge name='br0'/> </network> # brctl show bridge name bridge id STP enabled interfaces br0 8000.fc15b4137688 no eno1 vnet0 Both server1 and centos1 have IP addresses in the same subnet, and both are reachable with ping from every other host in my network. In both server1 and centos1, the openssh-server configuration in /etc/ssh/sshd_config is the default one, and has not been changed. When I ssh with Putty to the physical server server1 IP address, everything works as expected: I get a login prompt, I enter my password and I log in. However, when I use Putty to connect to the centos1 VM, I do not get a login prompt whatsoever. So I think there might be some issue in between the server1 physical interface and my centos VM. I used openssh-server in debug mode, so see where the ssh connection hangs, and here is what I get: [...] debug1: Server will not fork when running in debugging mode. debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8 debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips 26 Jan 2017 debug1: private host key #0: ssh-rsa SHA256:pEuFQsodwK+0PoRzbVRba1ahHLEpwp8DG2KGQmxOGJk debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:F6HrSNWZhYaU7LMweI+RBviqTCHcTYyMBGPDz5OjT4c debug1: private host key #2: ssh-ed25519 SHA256:aG3V6jjPHXUnNeavbxT/xozqrb5q3yWDkkAmXBCdnGk debug1: inetd sockets after dupping: 3, 3 Connection from x.x.x.181 port 49436 on x.x.x.115 port 22 debug1: Client protocol version 2.0; client software version PuTTY_Release_0.70 debug1: no match: PuTTY_Release_0.70 debug1: Local version string SSH-2.0-OpenSSH_7.4 debug1: Enabling compatibility mode for protocol 2.0 debug1: SELinux support enabled [preauth] debug1: permanently_set_uid: 74/74 [preauth] debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth] debug1: SSH2_MSG_KEXINIT sent [preauth] I tried with other windows based ssh clients (MobaXterm) and the same issue happens. I discussed this with people in the openssh mailing list, and they said this issue could most probably be caused by a path MTU/fragmentation problem... Then I moved my centos1 qcow2 image in another physical server called server2, with exactly the same hw specs and network connections, where I have installed an all-in-one OpenStack Pike. The network would be managed with neutron in this case, however I have configured neutron exactly so that the centos1 VM interface connects through a linux bridge (managed by neutron) to the server1 physical network interface, like in the libvirt case. # brctl show bridge name bridge id STP enabled interfaces brqa13eec69-a4 8000.0e7faabad6d4 no eno1 tap8cb53db0-fb tapb24a1cc5-20 Above the tapb24a1cc5-20 is the tap interface towards my centos1 VM. In this case, the Putty issue is gone, and I do not have any issue anymore. If I go back to the libvirt environment in server1, I get the same issue again. So I tend to think that my ssh connection issue is caused by the libvirt and the way networking is configured, however I do not know how to troubleshoot this further anymore. Any help is greatly appreciated. Adrian _______________________________________________ libvirt-users mailing list libvirt-users(a)redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users

3 5

[libvirt-users] How do i associate a VM to An IP
by Shashwat shagun 29 Jan '18

29 Jan '18

I want to block my VM with mac addr 52:54:00:a1:05:b6 from using IPs other than 192.168.124.130 is there any libvirt way to do this? -- Regards, Shashwat Shagun

3 2

[libvirt-users] Static DHCP assignments to VMs connected to bridge
by Dagmawi Biru 29 Jan '18

29 Jan '18

Hello libvirt-users, Given the following setup: --- [Rest of network/dhcp server] ------[ physical interface] ------ [ bridge] <----> | <---> [ VM ] I have an interface configured in my Domain XML like so: --- <interface type="bridge"> <source bridge="br20"> </interface> Bridge "br20" is connected to an external network that has a DHCP server. However, I want to set it so the interface for this VM has a static IP served to it, either by a locally running DHCP server or via some sort of static IP assignment in the XML configuration. Is this possible?

2 1

[libvirt-users] issue with openssh-server running in a libvirt based centos virtual machine
by Adrian Pascalau 27 Jan '18

27 Jan '18

Hi, I have a strange issue in a libvirt environment, and I do not know how to solve it. I have two centos hosts: first one is a physical server called server1, that acts as a host for the second one, called centos1. The centos1 is a virtual machine (VM) running in server1. A linux bridge in forwarding mode is used to connect the centos1 VM network interface to the server1 network interface and to the external network. The centos1 VM and the linux bridge are managed with libvirt (well, the bridge itself in this case is created manually). # virsh net-dumpxml br0 <network connections='1'> <name>br0</name> <uuid>5aaf72a5-023d-4b84-9d7c-d68b0918f620</uuid> <forward mode='bridge'/> <bridge name='br0'/> </network> # brctl show bridge name bridge id STP enabled interfaces br0 8000.fc15b4137688 no eno1 vnet0 Both server1 and centos1 have IP addresses in the same subnet, and both are reachable with ping from every other host in my network. In both server1 and centos1, the openssh-server configuration in /etc/ssh/sshd_config is the default one, and has not been changed. When I ssh with Putty to the physical server server1 IP address, everything works as expected: I get a login prompt, I enter my password and I log in. However, when I use Putty to connect to the centos1 VM, I do not get a login prompt whatsoever. So I think there might be some issue in between the server1 physical interface and my centos VM. I used openssh-server in debug mode, so see where the ssh connection hangs, and here is what I get: [...] debug1: Server will not fork when running in debugging mode. debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8 debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips 26 Jan 2017 debug1: private host key #0: ssh-rsa SHA256:pEuFQsodwK+0PoRzbVRba1ahHLEpwp8DG2KGQmxOGJk debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:F6HrSNWZhYaU7LMweI+RBviqTCHcTYyMBGPDz5OjT4c debug1: private host key #2: ssh-ed25519 SHA256:aG3V6jjPHXUnNeavbxT/xozqrb5q3yWDkkAmXBCdnGk debug1: inetd sockets after dupping: 3, 3 Connection from x.x.x.181 port 49436 on x.x.x.115 port 22 debug1: Client protocol version 2.0; client software version PuTTY_Release_0.70 debug1: no match: PuTTY_Release_0.70 debug1: Local version string SSH-2.0-OpenSSH_7.4 debug1: Enabling compatibility mode for protocol 2.0 debug1: SELinux support enabled [preauth] debug1: permanently_set_uid: 74/74 [preauth] debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth] debug1: SSH2_MSG_KEXINIT sent [preauth] I tried with other windows based ssh clients (MobaXterm) and the same issue happens. I discussed this with people in the openssh mailing list, and they said this issue could most probably be caused by a path MTU/fragmentation problem... Then I moved my centos1 qcow2 image in another physical server called server2, with exactly the same hw specs and network connections, where I have installed an all-in-one OpenStack Pike. The network would be managed with neutron in this case, however I have configured neutron exactly so that the centos1 VM interface connects through a linux bridge (managed by neutron) to the server1 physical network interface, like in the libvirt case. # brctl show bridge name bridge id STP enabled interfaces brqa13eec69-a4 8000.0e7faabad6d4 no eno1 tap8cb53db0-fb tapb24a1cc5-20 Above the tapb24a1cc5-20 is the tap interface towards my centos1 VM. In this case, the Putty issue is gone, and I do not have any issue anymore. If I go back to the libvirt environment in server1, I get the same issue again. So I tend to think that my ssh connection issue is caused by the libvirt and the way networking is configured, however I do not know how to troubleshoot this further anymore. Any help is greatly appreciated. Adrian

1 0

[libvirt-users] Create virtual machine failed using virDomainCreateXML
by netsurfed 25 Jan '18

25 Jan '18

Hi all, When I created a virtual machine using virDomainCreateXML, with virtualport type was openvswitch, and virtual machine creation failed. The error message is: internal error: Child process (/sbin/tc filter add dev vnet110 parent ffff: protocol all u32 match u32 0 0 police rate 0kbps burst 0kb mtu 64kb drop flowid :1) unexpected exit status 2: RTNETLINK answers: Invalid argument The Domain XML file the <interface> section like this: <interface type='bridge'><mac address='52:54:00:71:b1:b6'/><source bridge='ovsbr'/><virtualport type='openvswitch'/><address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/></interface> I looked at the system log and it looked like an ovs port problem: 4560 Jan 22 17:43:33 ubuntu-24 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exists del-port vnet110 -- add-port lcs_br_1 vnet110 -- set Interface vnet110 "external- ids:attached-mac=\"52:54:00:21:63:25\"" -- set Interface vnet110 "external-ids:iface-id=\"d0e5c770-6f03-4703-b107-3b074f989b06\"" -- set Interface vnet110 "external-ids:vm-id=\"1c79066e- 46a2-4fa0-984c-a88a7fffcd2b\"" -- set Interface vnet110 external-ids:iface-status=active 4561 Jan 22 17:43:33 ubuntu-24 kernel: [29028.418651] device vnet110 entered promiscuous mode 4562 Jan 22 17:43:33 ubuntu-24 libvirtd[4307]: 2018-01-22 09:43:33.507+0000: 4310: error : virCommandWait:2572 : internal error: Child process (/sbin/tc filter add dev vnet110 parent ffff: pr otocol all u32 match u32 0 0 police rate 0kbps burst 0kb mtu 64kb drop flowid :1) unexpected exit status 2: RTNETLINK answers: Invalid argument 4563 Jan 22 17:43:33 ubuntu-24 libvirtd[4307]: We have an error talking to the kernel 4564 Jan 22 17:43:33 ubuntu-24 kernel: [29028.454028] device vnet110 left promiscuous mode 4565 Jan 22 17:43:33 ubuntu-24 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=5 -- --if-exists del-port vnet110 However, when I use "tc" to manage "vnet*" of an existing virtual machine, it still returns the same error. root@ubuntu-24:~# /sbin/tc filter add dev vnet83 parent ffff: protocol all u32 match u32 0 0 police rate 0kbps burst 0kb mtu 64kb drop flowid :1 RTNETLINK answers: Invalid argument We have an error talking to the kernel Is there any reason for this problem? Thank you very much. Below some information about my machine: libvirt version: 3.4.0 ovs_version: "2.8.90" root@ubuntu-24:~# uname -a Linux ubuntu-24 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

2 3

[libvirt-users] Create virtual machine failed using virDomainCreateXML
by netsurfed 24 Jan '18

24 Jan '18

Hi all, When I created a virtual machine using virDomainCreateXML, with virtualport type was openvswitch, and virtual machine creation failed. The error message is: internal error: Child process (/sbin/tc filter add dev vnet110 parent ffff: protocol all u32 match u32 0 0 police rate 0kbps burst 0kb mtu 64kb drop flowid :1) unexpected exit status 2: RTNETLINK answers: Invalid argument The Domain XML file the <interface> section like this: I looked at the system log and it looked like an ovs port problem. However, when I use "tc" to manage "vnet*" of an existing virtual machine, it still returns the same error. root@ubuntu-24:~# /sbin/tc filter add dev vnet83 parent ffff: protocol all u32 match u32 0 0 police rate 0kbps burst 0kb mtu 64kb drop flowid :1 RTNETLINK answers: Invalid argument We have an error talking to the kernel Is there any reason for this problem? Thank you very much. Below some information about my machine: libvirt version: 3.4.0 ovs_version: "2.8.90" root@ubuntu-24:~# uname -a Linux ubuntu-24 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

1 0