- Users - libvirt List Archives

[libvirt-users] libvirt hooks deadlocks and auto-restarting a stopped guest
by Fred Clift 30 Mar '16

30 Mar '16

All, Short version: I want to auto-restart a particular guest any time it shuts down. libvirt hooks can't call 'virsh start foo'. What is a good/simple way to do this? Long version: I have what is probably an unusual set of requirements. To summarize briefly: I have an OS image that runs 3rd party binaries that I can't modify. These binaries occasionally get they system into a state that can not be resolved by restarting the software. We can detect the error state, but not prevent it or fix either the OS or the software. (The joys of closed-source software... sigh.) For a long time this has been running on real hardware, and a small monitor was written that will initiate a reboot of the system when it gets in that state. I ALSO use a custom qemu command line option: -snapshot so that I throw away all changes each time the system shuts down. These are all part of a compute cluster and no non-transient data is ever on these systems. What I'm trying to do: use snapshots AND on every OS boot, start fresh from the base image. It works if I give two custom options to qemu-kvm (-snapshot -no-reboot). Everytime I shut the system down, and then start it, I get back to the starting image. But if the error happens, I either shut down (with the -no-reboot option) and stay down, or I do a warm-boot and dont discard all filesystem changes. So this looked to me like a place to use libvirt hooks (Centos 7, I had to create /etc/libvirt/hooks and restart the libvirtd service). I made a qemu hook that watches for this dom name to get a release event so that I could auto-restart it. Of course you probably already know what I didn't - you can't call 'virsh start' safely from a libvirt hook. So, how can I make libvirt auto restart a dom? a libvirt hook doesn't seem to be the way. Several options occur: I might make the hook spawn something in the background that sleeps 5 seconds and then does a virsh start - presumably some time after the hook script exits. This seems somewhat problematic, and not reliable. I could make a cron-job that checks every minute and just restarts it. I'd like the restart to be faster than an average of 30 seconds though. I could replace the <emulator>/.....</emulator> in the dom definition with a script that wraps the real emulator in a loop, maybe... I could write an libvirt api consuming application that watches for the right events (reboot and shutdown) and then does the right thing. Other ideas? Fred Clift

1 0

Re: [libvirt-users] VM crash and lock manager
by Eric Blake 30 Mar '16

30 Mar '16

[moderator note: .pngs were stripped to avoid overwhelming the mail server and list recipients with 1M of data] -------- Forwarded Message -------- Date: Wed, 30 Mar 2016 11:25:09 +0200 Message-ID: <20160330112509.Horde.0Oad-ZfkzdZnouXf-VkEmw1(a)webmailperso.univ-brest.fr> From: villeneu(a)kassis.univ-brest.fr To: Franky Van Liedekerke <liedekef(a)telenet.be> Cc: libvirt-users(a)redhat.com Subject: Re: [libvirt-users] VM crash and lock manager References: <20160329192740.Horde.BZXrmDPzMBshASPvOs_x4Q6(a)webmailperso.univ-brest.fr> <20160329231311.1ae867a1(a)telenet.be> In-Reply-To: <20160329231311.1ae867a1(a)telenet.be> --- Franky Van Liedekerke <liedekef(a)telenet.be> a écrit : > On Tue, 29 Mar 2016 19:27:40 +0200 > villeneu(a)kassis.univ-brest.fr wrote: > >> Hello >> >> I changed my hypervisors ( 8) from centos 6.0 to fedora 23 and I >> added locks to prevent starting VM on multiple nodes, and it's work >> well. >> >> Since I changed the OS some old VMs ( centos 5.x and 6.x ) with very >> old kernel seem very instable and crash very often ( after one or two >> days ). Before they never have these problems. >> >> I saw, perhaps the reason is the dev is too slow ??? see screenshot >> joined >> >> A major problem with these crashs is that the VMs couldn't be >> destroyed by the virsh command, the qemu process is notified as >> defunct by the ps command. >> >> with virsh destroy VM >> I often get >> >> Failed to terminate process xxx with SIGTERM: Device or resource busy >> and the VM is still in the list >> >> If I try to remove all the file associated with the VM in /var/run or >> /var/lib....channel .. It doesn't give results The >> VM is still running as defunct and after a libvirt restart , libvirt >> and virsh command are pending. >> >> With sanlock it's a problem because the is never release and it's >> impossible to restart the VM. I can't remove the VM in the sanlock >> list. I tried kill commands, rm ... but without a real success. >> >> I tried many command with sanlock perhaps I misunderstood some >> commands but I never succeed in removing lock on the VMs. >> >> sanlock client rem_lockspace -r >> __LIBVIRT__DISKS__:b90b9c61e2d6413077205907ffb3281a:/var/lib/libvirt/images/POOL_ADMIN/sanlock/b90b9c61e2d6413077205907ffb3281a:0:4 >> -p 9383 >> >> So I'm trying to use lock_manager to replace the sanlock but >> lock_manager doesn't have tools to retrieve the lock / VM name in >> case of failure or crash. >> >> My questions are, is there a way to release or to force to release a >> lock with sanlock even if the qemu process is defunc and a way to >> restart libvirt in normal usage ? >> >> Perhaps by the API it`s possible to forece the release the lock ? >> >> Is there tools to retreive or calculate SHA256 lock for a VM with the >> lock_manager lock. I saw it's the lock is a SHA256 hash of the path >> and the VM name but I nerver get the good hash when I try by the hand. >> >> Is there tools to retreive or calculate SHA256 the lock for a VM with >> the lock_manager lock. I saw that the lock is a SHA256 hash of the >> path and the VM name but I never get the good hash when I try by the >> hand. >> >> Thanks. >> Sorry for my poor english ... I 'am french >> Michel > > It seems the lock held by sanlock isn't being released, so the vm can't > be shut down properly. Maybe some selinux issue? > I've been using virtlockd without troubles for more than a year already > (without selinux). For the locks, see if the lslocks command helps you. > > Franky > > _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.comhttps://www.redhat.com/mailman/listinfo/libvirt-users I do not use selinux too, I preffered sanlock at the beginning because sanlock comes with tools to remove locks. But when VMs crash abnormaly or freeze ( see the sreenshot I missed to attach it yesterday ) there is no way to unlease or remove the locks, they are presents even if I kill the process qemu and I can't restart the VM because the lock is still here. So I try now lock-manager, but do you know exactly how the lock is calculate . In case of problem I want to be able to retrieve the lock coupled with the VM. I have seen that it is a hash SHA256 of the path and the name, but if a try to do by the hand a sha256 of the complete path of the VM I don't have the correct value ( the same value as lock_manager ). Any Ideas ? Thanks crash1.png crash2.png

1 0

[libvirt-users] Hello and a question about QEMU log
by zhukaijie 30 Mar '16

30 Mar '16

Hello, Each running VM has it's command line logged in /var/log/libvirt/qemu/${vmname}. And now I'd like to disable this log function, that is to say, not to record the QEMU command line of VM. Could you please tell me will can I configure it? Best wishes.

2 1

[libvirt-users] VM crash and lock manager
by villeneu＠kassis.univ-brest.fr 29 Mar '16

29 Mar '16

Hello I changed my hypervisors ( 8) from centos 6.0 to fedora 23 and I added locks to prevent starting VM on multiple nodes, and it's work well. Since I changed the OS some old VMs ( centos 5.x and 6.x ) with very old kernel seem very instable and crash very often ( after one or two days ). Before they never have these problems. I saw, perhaps the reason is the dev is too slow ??? see screenshot joined A major problem with these crashs is that the VMs couldn't be destroyed by the virsh command, the qemu process is notified as defunct by the ps command. with virsh destroy VM I often get Failed to terminate process xxx with SIGTERM: Device or resource busy and the VM is still in the list If I try to remove all the file associated with the VM in /var/run or /var/lib....channel .. It doesn't give results The VM is still running as defunct and after a libvirt restart , libvirt and virsh command are pending. With sanlock it's a problem because the is never release and it's impossible to restart the VM. I can't remove the VM in the sanlock list. I tried kill commands, rm ... but without a real success. I tried many command with sanlock perhaps I misunderstood some commands but I never succeed in removing lock on the VMs. sanlock client rem_lockspace -r __LIBVIRT__DISKS__:b90b9c61e2d6413077205907ffb3281a:/var/lib/libvirt/images/POOL_ADMIN/sanlock/b90b9c61e2d6413077205907ffb3281a:0:4 -p 9383 So I'm trying to use lock_manager to replace the sanlock but lock_manager doesn't have tools to retrieve the lock / VM name in case of failure or crash. My questions are, is there a way to release or to force to release a lock with sanlock even if the qemu process is defunc and a way to restart libvirt in normal usage ? Perhaps by the API it`s possible to forece the release the lock ? Is there tools to retreive or calculate SHA256 lock for a VM with the lock_manager lock. I saw it's the lock is a SHA256 hash of the path and the VM name but I nerver get the good hash when I try by the hand. Is there tools to retreive or calculate SHA256 the lock for a VM with the lock_manager lock. I saw that the lock is a SHA256 hash of the path and the VM name but I never get the good hash when I try by the hand. Thanks. Sorry for my poor english ... I 'am french Michel

2 1

Re: [libvirt-users] "INTx fd" busy error on VM startup at boot, subsequent startup okay
by Pittman, Randall 29 Mar '16

29 Mar '16

Hey Mike, did you ever have any success figuring out this issue? I am having the same problem (on CentOS 7) with a custom data acquisition PCI card. It seems like the solution might have something to do with delaying the start of libvirtd in systemd until the PCI card (or vfio?) is ready, but I don’t know how to do that. Can anyone else offer assistance? My output: 2016-03-29T18:29:26.191010Z qemu-kvm: -device vfio-pci,host=05:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: Error: Failed to setup INTx fd: Device or resource busy 2016-03-29T18:29:26.289154Z qemu-kvm: -device vfio-pci,host=05:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device initialization failed. 2016-03-29T18:29:26.289195Z qemu-kvm: -device vfio-pci,host=05:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device 'vfio-pci' could not be initialized 2016-03-29 18:29:26.337+0000: shutting down Thanks, Randy <--Original Message--> Hey all, I'm getting an error starting a libvirt managed qemu/kvm VM at physical host boot time, but manually starting it afterwards works fine. This is on a Ubuntu Wily i7-4790 box running Linux 4.2 and libvirt 1.2.16. There is a legacy (5V) PCI card being passed through to the VM, the error seems to relate to that. The error that always appears at boot in `/var/log/libvirt/qemu/test-vm.log` is: 2016-01-11T05:29:28.487935Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,id=hostdev0,bus=pci.0,addr=0x4: vfio: Error: Failed to setup INTx fd: Device or resource busy 2016-01-11T05:29:28.488148Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,id=hostdev0,bus=pci.0,addr=0x4: Device initialization failed 2016-01-11T05:29:28.488169Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,id=hostdev0,bus=pci.0,addr=0x4: Device 'vfio-pci' could not be initialized After the physical machine has completed booting, starting the VM via virsh or virt-manager works fine. The card is an old Firewire card. I have blacklisted the firewire modules on the physical machine so they aren't loaded, but is it possible that the kernel, udev or some other process still has access to the device when systemd starts the libvirt-bin service? Any clues greatly appreciated. Thanks, //Mike -- ⊨ Michael Gratton, Percept Wrangler. ⚙ <http://mjog.vee.net/>

1 0

[libvirt-users] Using Openvswitch and qemu:///session
by Predrag Ivanovic 24 Mar '16

24 Mar '16

Hi. I'd like to use Openvswitch while running libvirt as an unprivileged user (qemu:///session). As expected, system session works just fine with OVS. When I try to start the domain from user session, with the relevant network part of the domain xml edited to use openvswitch, like this --- <interface type='bridge'> <mac address='52:54:00:4a:ef:bb'/> <source bridge='ovs-br'/> <virtualport type='openvswitch'> <parameters interfaceid='33ec3d9d-a92a-468e-aa48-6c31f7882d6f'/> </virtualport> <target dev='veth0-ovs'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> --- I get this error: --- start freebsd-10.3 error: Failed to start domain freebsd-10.3 error: internal error: /usr/lib/qemu/qemu-bridge-helper --use-vnet --br=ovs-br --fd=19: failed to communicate with bridge helper: Transport endpoint is not connected stderr=failed to add interface `tap0' to bridge `ovs-br': Operation not supported --- Now, if I understand correctly, this is Qemu, not libvirt error? Libvirt prepares the network, passes it to qemu-bridge-helper, which has no idea what to do with it, since it's not the regular bridge, and barfs? The reason I ask is because I saw calls to ovs-vsctl in the logs when I did the same thing using qemu:///system. My question (finally): if I reconfigure OVS to not run as root:root, but as ovs:ovs, and add the user to that group, will libvirt/virsh automagically figure out that the user has permissions to run ovs-whatever and write permission to /var/run/openvswitch/db.sock, so there is no need for qemu-bridge-helper, and just set up the network, just like when system sesion is used, or is using qemu-bridge-helper the only option available for non-root users ? If it is, due to some Linux constraints, I'll curse a bit and migrate the domains :) And, before I forget, I am using libvirt and qemu built from git master, and openvswitch-2.5.0 Pedja

2 2

[libvirt-users] virsh and bash_complete
by Mohsen Pahlevanzadeh 24 Mar '16

24 Mar '16

3 3

[libvirt-users] /run/user/<uid> persists after user account is deleted
by Lars Kellogg-Stedman 24 Mar '16

24 Mar '16

In the tripleo-quickstart project, we're making use of libvirt's unprivileged mode. When run as a non-root user using a qemu:///session uri, libvirt creates files and sockets in subdirectories of $XDG_RUNTIME_DIR (aka /run/user/<uid>). We are seeing situations in which: - A "stack" user is created - Some libvirt resources (a storage pool) are provisioned for that user - The "stack" user is deleted (we first kill off any processes owned by the stack user, then delete the user via ansible which ultimately calls "userdel -r stack"). - The corresponding /run/user/<uid> directory is *not* deleted - A new "stack" user is created, and the stale storage pool from the previous invocation exists in an indeterminate state such at attempts to undefine it fail with: storage pool the_pool exists but directory /home/stack/pool is missing I can work around the problem like this: virsh pool-dumpxml the_pool | virsh pool-define /dev/stdin This ensures that a subsequent pool-undefine operation can successfully delete the pool. I think there are two problems here. Once is clearly a system problem; the /run/user directory should be deleted when the user account is deleted, period. The other problem seems like a libvirt problem: faced with a storage pool in this state, libvirt should permit the pool to be undefined anyway. It's possible this is already fixed in later releases of libvirt, but this is on CentOS 7, so we're stuck with libvirt-1.2.17-13.el7_2.3.x86_64 for now. I'm curious if anyone has seen this behavior or knows whether or not it has been corrected in later releases. Thanks, -- Lars Kellogg-Stedman <lars(a)redhat.com> | larsks @ {freenode,twitter,github} Cloud Engineering / OpenStack | http://blog.oddbit.com/

1 0

[libvirt-users] libvirt 1.3.3 and Hyper-V 2012
by Schwab Michael 23 Mar '16

23 Mar '16

Hello I installed libvirt 1.3.3 for managing a Hyper-V 2012 R2 Server. On the hyper-v server I entered the two commands: winrm set winrm/config/service/auth @{Basic="true"} winrm set winrm/config/service @{AllowUnencrypted="true"} After connecting, I receive the following error: user@ubuntu-serv-kvm:~/libvirt-hyperv-r2-2012$ virsh -c hyperv://192.168.33.140/?transport=http Enter username for 192.168.33.140 [administrator]: Enter administrator's password for 192.168.33.140: error: failed to connect to the hypervisor error: internal error: SOAP fault during enumeration: code 's:Sender', subcode 'n:CannotProcessFilter', reason 'The data source could not process the filter. The filter might be missing or it might be invalid. Change the filter and try the request again. ', detail '500The specified class does not exist in the given namespace. HRESULT 0x8033801a0052150858778HRESULTThe specified class does not exist in the given namespace. ' I found a patch: https://github.com/vikhyath/libvirt-hyperv-r2-2012 How can I add this patch? exist another method? Thanks for your help Greatings Michael

1 0

[libvirt-users] Virtual Netwok manager and cmd
by Mohsen Pahlevanzadeh 23 Mar '16

23 Mar '16

Dear All, I'm reading the following pages: http://wiki.libvirt.org/page/TaskRoutedNetworkSetupVirtManager http://wiki.libvirt.org/page/TaskNATSetupVirtManager http://wiki.libvirt.org/page/TaskIsolatedNetworkSetupVirtManager routed mode, NAT mode and Isolated mode, But above pages explain with GUI, a sysadmin forgets GUI... Question is What's replacement of above command? ---Best Regards Mohsen

2 1