May 2018 - Users - libvirt List Archives

[libvirt-users] Two Node Cluster
by Cobin Bluth 01 Jun '18

01 Jun '18

Hello Libvirt Users, I would like to setup a two node bare-metal cluster. I need to guidance on the network configuration. I have attached a small diagram, the same diagram can be seen here: https://i.imgur.com/SOk6a6G.png I would like to configure the following details: - Each node has a DHCP enabled guest network where VMs will run. (eg, *192.168.1.0/24 <http://192.168.1.0/24>* for Host1, and *192.168.2.0/24 <http://192.168.2.0/24>* for Host2) - Any guest in Host1 should be able to ping guests in Host2, and vice versa. - All guests have routes to reach the open internet (so that '*yum update*' will work "out-of-the-box") - Each node will be able to operate fully if the other physical node fails. (no central DHCP server, etc) - I would like to *add more physical nodes later* when I need the resources. This is what I have done so far: - Installed latest Ubuntu 18.04, with latest version of libvirt and supporting software from ubuntu's apt repo. - Each node can reach the other via its own eth0. - Each node has a working vxlan0, which can ping the other via its vxlan0, so it looks like the vxlan config is working. (I used *ip link add vxlan0 type vxlan...*) - Configured route on Host1 like so: *ip route add 192.168.2.0/24 <http://192.168.2.0/24> via 172.20.0.1* - Configured route on Host2 also: *ip route add 192.168.1.0/24 <http://192.168.1.0/24> via 172.20.0.2* - All guests on Host1 (and Host1) can ping eth0 and vxlan0 on Host2, and vice versa, yay. - Guests on Host1 *cannot* ping guests on Host2, I suspect because the the default NAT config of the libvirt network. So, at this point I started to search for tutorials or more information/documentation, but I am a little overwhelmed by the sheer amount of information, as well as a lot of "stale" information on blogs etc. I have learned that I can *virsh net-edit default*, and then change it to an "open" network:* <forward mode='open'/>* After doing this, the guests cannot reach outside their own network, nor reach the internet, so I assume that I would need to add some routes, or something else to get the network functioning like I want it. There is also *<forward mode="route"/>*, but I dont fully understand the scenarios where one would need an *open* or a *route* forward mode. I have also shied away from using openvswitch, and have opted for ifupdown2. (I have taken most of my inspiration from this blog post: https://joejulian.name/post/how-to-configure-linux-vxlans-with-multiple-uni… ) Some questions that I have for the mailing list, any help would be greatly appreciated: - Is my target configuration of a KVM cluster uncommon? Do you see drawbacks of this setup, or does it go against "typical convention"? - Would my scenario be better suited for an "*open*" network or a "*route*" network? - What would be the approach to complete this setup?

3 4

Re: [libvirt-users] VM I/O performance drops dramatically during storage migration with drive-mirror
by Kashyap Chamarthy 29 May '18

29 May '18

Cc the QEMU Block Layer mailing list (qemu-block(a)nongnu.org) who might have more insights here; and wrap long lines. On Mon, May 28, 2018 at 06:07:51PM +0800, Chunguang Li wrote: > Hi, everyone. > > Recently I am doing some tests on the VM storage+memory migration with > KVM/QEMU/libvirt. I use the following migrate command through virsh: > "virsh migrate --live --copy-storage-all --verbose vm1 > qemu+ssh://192.168.1.91/system tcp://192.168.1.91". I have checked the > libvirt debug output, and make sure that the drive-mirror + NBD > migration method is used. > > Inside the VM, I use an I/O benchmark (Iometer) to generate an oltp > workload. I record the I/O performance (IOPS) before/during/after > migration. When the migration begins, the IOPS dropped by 30%-40%. > This is reasonable, because the migration I/O competes with the > workload I/O. However, during almost the last period of migration > (which is 66s in my case), the IOPS dropped dramatically, from about > 170 to less than 10. I also show the figure of this experiment in the > attachment of this email. [The attachment should arrive on the 'libvirt-users' list archives; but it's not there yet -- https://www.redhat.com/archives/libvirt-users/2018-May/thread.html] > I want to figure out what results in this period with very low IOPS. > First, I added some printf()s in the QEMU code, and knew that, this > period occurs just before the memory migration phase. (BTW, the memory > migration is very fast, which is just about 5s.) So I think this > period should be the last phase of the "drive-mirror" process of QEMU. > So then I tried to read the code of "drive-mirror" in QEMU, but failed > to understand it very well. > > Does anybody know what may lead to this period with very low IOPS? > Thank you very much. > > Some details of this experiment: The VM disk image file is 30GB > (format = raw,cache=none,aio=native), and Iometer operates on an 10GB > file inside the VM. The oltp workload consists of 33% writes and 67% > reads (8KB request size, all random). The VM memory size is 4GB, most > of which should be zero pages, so the memory migration is very fast. > > -- > Chunguang Li, Ph.D. Candidate > Wuhan National Laboratory for Optoelectronics (WNLO) > Huazhong University of Science & Technology (HUST) > Wuhan, Hubei Prov., China -- /kashyap

3 4

[libvirt-users] Using libvirt to access Ceph RBDs with Xen
by thg 29 May '18

29 May '18

Hi everybody, my background: I'm doing Xen since 10++ years, many years with DRBD for high availability, since some time I'm using preferable GlusterFS with FUSE as replicated storage, where I place the image-files for the vms. In my current project we started (successfully) with Xen/GlusterFS too, but because the provider, where we placed the servers, uses widely CEPH, we decided to switch too. Unfortunately I'm new to CEPH, but with help of a technician, we have running a 3 node CEPH-cluster now, that seems to work fine. Hardware: - Xeons, 24 Cores, 256 GB RAM, 2x 240 GB system-SSDs RAID1, 4x 1,92 TB data-SSDs (no RAID) Software we are using: - CentOS 7.5.1804 - Kernel: 4.9.86-30.el7 @centos-virt-xen-48 - Xen: 4.8.3-5.el7 @centos-virt-xen-48 - libvirt-xen: 4.1.0-2.xen48.el7 @centos-virt-xen-48 - Ceph: 2:12.2.5-0.el7 @Ceph CEPH itself works, Xen too and even libvirt (I'm quite new in this too, normally I use the XL-tools to manage my VMs) ;-) I can manually map an RBD and start a PV-VM via libvirt, after modifying the XML-config and defined the VM with libvirt. That is what happens: # virsh define xml/testvm3.xml Domain testvm3 defined from xml/testvm3.xml # virsh start --console testvm3 error: Failed to start domain testvm3 error: internal error: libxenlight failed to create new domain 'testvm3' xml/testvm3.xml: -> ... <devices> <disk type='network' device='disk'> <source protocol='rbd' name='devel-pool/testvm3.rbd'> <host name="10.20.30.1" port="6789"/> <host name="10.20.30.2" port="6789"/> <host name="10.20.30.3" port="6789"/> </source> <auth username='libvirt'> <secret type='ceph' uuid='07f3a0fe-9178-xxxxxxxxx'/> </auth> <target dev='xvda' bus='xen'/> </disk> ... Logs: /var/log/libvirt/libxl/libxl-driver.log: -> ... 2018-05-27 11:24:36.666+0000: libxl: libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult logfile /var/log/xen/bootloader.14.log 2018-05-27 11:24:36.666+0000: libxl: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [183216] exited with error status 1 2018-05-27 11:24:36.667+0000: libxl: libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3 /var/log/xen/bootloader.14.log: -> Traceback (most recent call last): File "/usr/lib64/xen/bin/pygrub", line 896, in <module> part_offs = get_partition_offsets(file) File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets image_type = identify_disk_image(file) File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image fd = os.open(file, os.O_RDONLY) OSError: [Errno 2] No such file or directory: 'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789' So the Xen-Python-script, that searches for the kernel/bootloader of the VM, does not find the imagefile. But if I grab the above output and use it with qemu, it does find the blockdevice: # qemu-img info "rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789" -> image: rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxxxxxxxxxxxxxx==:auth_supported=cephx\;none:mon_host=10.20.30.1\:6789\;10.20.30.2\:6789\;10.20.30.3\:6789 file format: raw virtual size: 16G (17179869184 bytes) disk size: unavailable Thus it seems, that CEPH-auth works and the VM-XML-config is interpreted correct by libvirt. But the next step fails, when libvirt should pass the blockdevice to Xen? Anybody out there who uses a similar configuration? Any hints how to debug this? Unfortunately I do not have anymore logs to look at. Thank you very much, -- Kind regards, thg

2 1

[libvirt-users] virsh migration job unexpectedly failed
by cokabug 26 May '18

26 May '18

Hi , when i have run virsh migrate command , the migration progress fiailed at [61 %]. command : virsh migrate --live --undefinesource --p2p --copy-storage-inc --verbose --persistent instance-00001959 qemu+tcp://host292/system result : Migration: [ 61 %]error: operation failed: migration job: unexpectedly failed os/kernel version: centos 6.7 / kernel 2.6.32 qemu-kvm version : 0.12.1.2 libvirt version : 0.10.2-54 the libvirt.log on target host: 2018-05-22 11:45:38.603+0000: 32314: warning : qemuDomainObjTaint:1459 : Domain id=30 name='instance-00001959' uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 is tainted: high-privileges 2018-05-22 11:45:38.883+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:38.902+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:38.903+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.090+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.090+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:39.091+0000: 32314: warning : qemuDomainObjEnterMonitorInternal:1062 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2018-05-22 11:45:41.841+0000: 32310: error : qemuMonitorIO:615 : internal error End of file from monitor the qmeu-kvminstance-00001959.log: 2018-05-22 11:45:38.603+0000: starting up LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/home/ning.zhao USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name instance-00001959 -S -M rhel6.6.0 -cpu Broadwell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+m$ nitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 41975d30-2490-473e-be7b-9d2d08bf66d2 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.1.2-4.el6,serial=fcbe6c27-c9$ d-e811-ac36-a281f9ebc503,uuid=41975d30-2490-473e-be7b-9d2d08bf66d2 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00001959.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -n$ -kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/41975d30-2490-473e-be7b-9d2d08bf66d2/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4$ drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:c0:8f:6b,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/41975d30-2490-473e-b$ 7b-9d2d08bf66d2/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 127.0.0.1:2 -k en-us -vga cirrus -incoming tcp: 0.0.0.0:49152 -device virtio-bal$ oon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on Domain id=30 is tainted: high-privileges char device redirected to /dev/pts/2 Receiving block device images qemu: warning: error while loading state for instance 0x0 of device 'block' load of migration failed 2018-05-22 11:45:41.841+0000: shutting down i checked the log and disk file and they have right permissions, has anyone know about this ,i have google many times ,but i cant find the right debug way to resolve it. thanks

2 2

[libvirt-users] Create qcow2 v3 volumes via libvirt
by Gionatan Danti 24 May '18

24 May '18

Hi all, on a fully patched CentOS 7.4 x86-64, I see the following behavior: - when creating a new volumes using vol-create-as, the resulting file is a qcow2 version 2 (compat=0.10) file. Example: [root@gdanti-lenovo vmimages]# virsh vol-create-as default zzz.qcow2 8589934592 --format=qcow2 --backing-vol /mnt/vmimages/centos6.img Vol zzz.qcow2 created [root@gdanti-lenovo vmimages]# file zzz.qcow2 zzz.qcow2: QEMU QCOW Image (v2), has backing file (path /mnt/vmimages/centos6.img), 8589934592 bytes [root@gdanti-lenovo vmimages]# qemu-img info zzz.qcow2 image: zzz.qcow2 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 196K cluster_size: 65536 backing file: /mnt/vmimages/centos6.img backing file format: raw Format specific information: compat: 0.10 refcount bits: 16 - when creating a snapshot, the resulting file is a qcow2 version 3 (comapt=1.1) file. Example: [root@gdanti-lenovo vmimages]# virsh snapshot-create-as centos6left --disk-only --no-metadata snap.qcow2 Domain snapshot snap.qcow2 created [root@gdanti-lenovo vmimages]# file centos6left.snap.qcow2 centos6left.snap.qcow2: QEMU QCOW Image (v3), has backing file (path /mnt/vmimages/centos6left.qcow2), 8589934592 bytes [root@gdanti-lenovo vmimages]# qemu-img info centos6left.snap.qcow2 image: centos6left.snap.qcow2 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 196K cluster_size: 65536 backing file: /mnt/vmimages/centos6left.qcow2 backing file format: qcow2 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false From what I know, this is a deliberate decision: compat=1.1 requires relatively recent qemu version, and creating a new volume play on the "safe side" of compatibility. It is possible to create a new volume using qcow2 version 3 (compat=1.1) format *using libvirt/virsh* (I know I can do that via qemu-img)? Any drawback on using version 3 format? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti(a)assyoma.it - info(a)assyoma.it GPG public key ID: FF5F32A8

6 17

[libvirt-users] Unable to connect VMs to a bridge over bonded network on Debian 9 (works fine on Centos 7.4)
by Thing 23 May '18

23 May '18

Hi, Sorry of this is the wrong list, trying to find some help. I am trying to set up 2 KVM hosts using Debian 9 (connecting to a 3 node Centos 7.4 gluster cluster) To get the bandwidth I have bonded 2 NICS each end. However while the network works fine for the Debian VM host the VM guests cannot connect/get an IP using DHCP connecting to br0. (Same setup on the Centos end works fine however) On the Debian host if I connect to br1 which is a bridge to a NIC port it works fine. My problem then seems to be at the bonding layer for libvirt/kvm ? Here is my interfaces config, I cannot see the problem. The VM guest is on the gluster cluster and boots fine just no networking. any ideas what I am doing wrong pls? or a better group/list to go ask in? auto eno1 iface eno1 inet manual #iface eno1 inet static bond-master bond0 #address 192.168.1.62 #netmask 255.255.255.0 #gateway 192.168.1.1 #dns-nameservers 192.168.1.241 192.168.1.104 # mtu 9000 auto enp1s0f1 iface enp1s0f1 inet manual bond-master bond0 # mtu 9000 auto bond0 iface bond0 inet static bond-miimon 100 bond-mode 6 bond-updelay 200 bond-xmit_hash_policy layer3+4 bond-lacp-rate 1 # mtu 9000 # address 192.168.1.62 # netmask 255.255.255.0 # gateway 192.168.1.1 # dns-nameservers 192.168.1.241 192.168.1.104 slaves eno1 enp1s0f1 auto br0 iface br0 inet static address 192.168.1.62 netmask 255.255.255.0 gateway 192.168.1.1 dns-nameservers 192.168.1.241 192.168.1.104 bridge_ports bond0 #bridge_ports eno1 enp1s0f1 bridge_stp on bridge_fd 0.0 # mtu 9000 #post-up ifconfig bond0 mtu 9000 #post-up ip link set dev bond0 mtu 9000 #auto bond1 #iface bond1 inet static # address 10.100.200.62 # netmask 255.255.255.0 # bond_slaves enp3s0f1 # bond_primary enp3s0f1 # mtu 9000 auto br1 iface br1 inet static address 10.100.200.62 netmask 255.255.255.0 gateway 10.100.200.1 bridge_ports enp3s0f1 bridge_stp on bridge_fd 0.0 root@kvm02:/etc/network#

1 0

[libvirt-users] Attempting to use Secure Boot for Windows 10
by frizop f 22 May '18

22 May '18

Anybody have a guide on using secure boot? I managed to set */usr/share/edk2/ovmf/OVMF_CODE.secboot.fd* as the* <loader readonly='yes' type='pflash'>*, but when I change this windows only shows me a black screen. I've also tried to fiddle with the following two settings: *<type arch='x86_64' machine='pc-q35-2.10'>* I'm running fedora 27. Any advice on how to implement secure boot for a VM under KVM/Virt-Manager/qemu/etc etc. I found this https://en.opensuse.org/openSUSE:UEFI_Secure_boot_using_qemu-kvm but it doesn't seem to match up with what I'm seeing on a few different aspects, (obviously of which, is I'm not using SuSE) Anyway, any help at all on this would be greatly appreciated. Thanks! -frizop

1 0

[libvirt-users] Performance of guest
by Allence 22 May '18

22 May '18

I recently used Linux in VMware, but I discovered that kvm can replace it, so I created the kvm environment and then started a window on the host system. But when I use it, I have a problem. My guest's performance is not very good. For example, when I use Windows without kvm, the “lol” fps is 160, but my guest is only 135. why? I only have one guest what is the reason? What can we do to promote my performance? There are my some settings for me. timer: hypervclock cpu: 8 socket(intel@ core i7 7700k) memory: 16G GPU: 1060 network: virtio my environment: libvirt: 4.2 qemu: 2.12 windows: windows 7 host: ubuntu server 1604 If you need my detailed settings, please let me know.

2 2

[libvirt-users] Guest's CPU frequency cannot be similar to host system
by Allence 18 May '18

18 May '18

My resource: kernel: 4.16.8 host system: Ubuntu 16.04 Llbvirt : 4.2 Qemu : 2.12 Cpu : intel@core i7 6700 guest system: windows7 professional My problem: In my host, Cpu frequency is 3800~3900 MHz(4 socket and 2 thread) because i turn on “Intel Turbo Boost”. And silent frequency is 3.40 GHz. Then i boot my guest, But in Guest my cpu frequency is 3408 MHz. why it's not 3800 MHz? And here is my guest xml: <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='8' threads='1'/> </cpu>

2 2

[libvirt-users] changing pci addr of SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
by daggs 12 May '18

12 May '18

Greetings, when execute lspci inside my vm linux guest, I see this: :/# lspci | grep -i scsi 03:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01) I don't see any such address in my xml file, is it possible to move that controller to another pci addr? Thanks, Dagg.

2 3