- Users - libvirt List Archives

Re: virsh not connecting to libvertd ?
by Jerry Buburuz 16 Jun '23

16 Jun '23

Michal PrÃvoznÃk > On 6/12/23 20:17, Jerry Buburuz wrote: >> Just found my issue. >> After I removed the cephfs mounts it worked! >> I will debug ceph. >> I assumed because I could touch files on mounted cephfs it was working. Now virsh list works! > Out of curiosity. Do you perhaps have a storage pool defined over cephfs? I can see two possible sources for the problem: > 1) autostarted storage pool that makes libvirt mount cephfs, or My storage is hard mounted in fstab. This works and it does mount on boot. # fstab on hypervisor user@.mynamefs-01=/ /data ceph noatime,_netdev 0 0 usern@.mynamefs-02=/ /data2 ceph noatime,_netdev 0 0 > 2) a storage pool defined over a path where cephfs is mounted. > The problem with 1) is obvious (in fact it's not specific to ceph, if it was NFS/iSCSI and the server wasn't responding then libvirtd would just hang). I agree with NFS/cephfs or any storage over the network if its not available libvirtd defined pools will cause problems. > The problem with 2) is that for some types of storage pools ('dir' typically) libvirt assumes they are always 'running'. And proceeds to enumerate volumes in that pool (i.e. files under the dir). And if there's a stale mount point, this might stuck libvirtd. But again, this is not limited to ceph, any network FS might do this. > Michal In my case I built this hypervisor using cephfs as my primary storage for virtualmachines over the past year. Its has worked until recently. Recently I had issue with my ceph which likely caused a stale mount. In the past if the storage went offline, after fixing the issue I had no problems and my virualmachines came back to life. In the current case I found: * cephfs working as normal. healthy. * hypervisors mounting using fstab as usual. * libvirtd starts normally no errors (even with cephfs mounted) * virsh fails to connect to libvirt when the cephfs is mounted. If I umount /cephfs, and systemctl restart libvirtd. virsh works! Example "virsh list" , "virsh version" ..etc. I am going to try and maybe delete the iso pools I have. One interesting thing I found yesterday after I restart the hypervisor and try umount /cephfs, libvirtd has one of my pools locked and opened. Its the pool with iso in it. I know someone in a previous response to me mentioned they had issues with iso being stored on cephfs and libvirtd. In order for me to umount /cephfs I have to stop all libvirtd services(livrtd, libvirtd.socket .. ro .. and admin). This makes sense libvirtd has defined pools in my cephfs. thanks jerry

1 0

virsh not connecting to libvertd ?
by Jerry Buburuz 16 Jun '23

16 Jun '23

I have identical two hypervisors same operating system: Ubuntu 22.04.2 LTS Recently both virsh stopped talking to the libvirtd. Both stopped within a few days of each other. Currently if I run: virsh uri virsh version virsh list # virsh list ..nothing just hangs When I ran strace on these broken machines it get stuck at same spot: strace virsh list ... access("/var/run/libvirt/virtqemud-sock", F_OK) = -1 ENOENT (No such file or directory) access("/var/run/libvirt/libvirt-sock", F_OK) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 5 connect(5, {sa_family=AF_UNIX, sun_path="/var/run/libvirt/libvirt-sock"}, 110) = 0 getsockname(5, {sa_family=AF_UNIX}, [128 => 2]) = 0 futex(0x7fa716a672f0, FUTEX_WAKE_PRIVATE, 2147483647) = 0 fcntl(5, F_GETFD) = 0 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fcntl(5, F_GETFL) = 0x2 (flags O_RDWR) fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 futex(0x7fa716a67348, FUTEX_WAKE_PRIVATE, 2147483647) = 0 eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK) = 6 write(6, "\1\0\0\0\0\0\0\0", 8) = 8 write(4, "\1\0\0\0\0\0\0\0", 8) = 8 write(4, "\1\0\0\0\0\0\0\0", 8) = 8 futex(0x7fa70c001cb0, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x7fa716a6786c, FUTEX_WAKE_PRIVATE, 2147483647) = 0 futex(0x7fa716a67378, FUTEX_WAKE_PRIVATE, 2147483647) = 0 write(4, "\1\0\0\0\0\0\0\0", 8) = 8 futex(0x7fa70c001cb0, FUTEX_WAKE_PRIVATE, 1) = 1 write(6, "\1\0\0\0\0\0\0\0", 8) = 8 rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0 poll([{fd=5, events=POLLOUT}, {fd=6, events=POLLIN}], 2, -1) = 2 ([{fd=5, revents=POLLOUT}, {fd=6, revents=POLLIN}]) read(6, "\2\0\0\0\0\0\0\0", 16) = 8 write(6, "\1\0\0\0\0\0\0\0", 8) = 8 write(6, "\1\0\0\0\0\0\0\0", 8) = 8 futex(0x5628ce6e9710, FUTEX_WAKE_PRIVATE, 2147483647) = 0 write(6, "\1\0\0\0\0\0\0\0", 8) = 8 write(6, "\1\0\0\0\0\0\0\0", 8) = 8 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 write(5, "\0\0\0\34 \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 28) = 28 write(6, "\1\0\0\0\0\0\0\0", 8) = 8 rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0 poll([{fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 2, -1) = 1 ([{fd=6, revents=POLLIN}]) read(6, "\5\0\0\0\0\0\0\0", 16) = 8 poll([{fd=5, events=POLLIN}, {fd=6, events=POLLIN}], 2, -1 It gets stuck at this poll(). Note I tested strace on identical new install of ubtuntu 22.04 where virsh connects fine and get identical strace, except after this poll() it continues on with read/write ..etc. I turned on debugging for libvirtd and get no errors while virsh is trying to connect. I am able to get a virsh# shell. The shell only hangs when I try "connect, uri, version". Another method of debugging I tried was: LIBVIRT_DEBUG=error LIBVIRT_LOG_FILTERS="1:* " virsh uri .. .. 2023-06-06 20:51:22.312+0000: 1647: debug : doRemoteOpen:1128 : Trying authentication 2023-06-06 20:51:22.312+0000: 1647: debug : virNetMessageNew:44 : msg=0x55b996539680 tracked=0 2023-06-06 20:51:22.312+0000: 1647: debug : virNetMessageEncodePayload:383 : Encode length as 28 2023-06-06 20:51:22.312+0000: 1647: info : virNetClientSendInternal:2151 : RPC_CLIENT_MSG_TX_QUEUE: client=0x55b996538010 len=28 prog=536903814 vers=1 proc=66 type=0 status=0 serial=0 2023-06-06 20:51:22.312+0000: 1647: debug : virNetClientCallNew:2107 : New call 0x55b996535f80: msg=0x55b996539680, expectReply=1, nonBlock=0 2023-06-06 20:51:22.312+0000: 1647: debug : virNetClientIO:1920 : Outgoing message prog=536903814 version=1 serial=0 proc=66 type=0 length=28 dispatch=(nil) 2023-06-06 20:51:22.312+0000: 1647: debug : virNetClientIO:1978 : We have the buck head=0x55b996535f80 call=0x55b996535f80 2023-06-06 20:51:22.312+0000: 1647: info : virEventGLibHandleUpdate:195 : EVENT_GLIB_UPDATE_HANDLE: watch=1 events=0 2023-06-06 20:51:22.312+0000: 1647: debug : virEventGLibHandleUpdate:206 : Update handle data=0x55b996534d30 watch=1 fd=5 events=0 2023-06-06 20:51:22.312+0000: 1647: debug : virEventGLibHandleUpdate:229 : Removed old handle source=0x55b996534de0 2023-06-06 20:51:22.312+0000: 1648: debug : virEventRunDefaultImpl:341 : running default event implementation Any help would be appreciated. thanks jerry

5 7

site certificate
by Daan Hoogland 13 Jun '23

13 Jun '23

LS, I am sure you have already noticed but I want to report it anyway; When compiling a dependend on "https://libvirt.org/maven2/net/juniper/tungsten/juniper-tungsten-api/2.0/ju…", I get a certificate error. I get the a similar error when accessing https://libvirt.org btw hope you can do something with this information, -- Daan

2 1

Criteria used to allow or not live migration
by Gianluca Cecchi 02 Jun '23

02 Jun '23

Hello, I would like to know what are the criteria that allow or not live migration when source and dest hosts have same CPU type (eg Intel) but different models (eg cascade lake vs ice lake). I imagine it is libvirtd to drive this decision, correct? Does libvirt has a sort of table or list of cpu flags that are mandatory / optional? Or how is it established to prevent or not live migration across different couodels systems? Thanks, Gianluca

1 0

issue about libvirtd: daemon can not start
by 王东亚 30 May '23

30 May '23

hi, I installed kmv on ubuntu-22.04-desktop-amd64, after installing some necessary package (qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virtinst), the libvirtd can not get start, actually, when I running systemctl start libvirtd.service, it'll run a little while (about 3 sec) and then it stops, or should I say it never start. I checked the status, it says ExecStart=/user/sbin/libvirtd $LIBVIRTD_ARGS (code=dumped, signal=TRAP) I processed journalctl -p err, it says ../../../glib/gmem.c:167 failed to allocate xxxxxxx bytes my laptop has 8G phycial memory any suggestion will be appricate. Best Regards.

2 1

Question regaring correct usage of CPU shares
by mailing-libvirt＠posteo.de 26 May '23

26 May '23

Hi there, I have a question regarding the shares option of the cputune section. I want to illustrate my question with the following example. Let's assume I have two virtual machines like the following on four dedicated core with two threads each: VM1: <cputune> <shares>512</shares> <vcpupin vcpu="0" cpuset="0"/> <vcpupin vcpu="1" cpuset="4"/> </cputune> VM2: cputune> <shares>512</shares> <vcpupin vcpu="0" cpuset="1"/> <vcpupin vcpu="1" cpuset="5"/> <vcpupin vcpu="2" cpuset="2"/> <vcpupin vcpu="3" cpuset="6"/> <vcpupin vcpu="4" cpuset="3"/> <vcpupin vcpu="5" cpuset="7"/> </cputune> Those two virtual machines I just mentioned won't interfere with one another since they are pinned to different cores. However I introduce another, third virtual machine, like this: VM3: <cputune> <shares>512</shares> <vcpupin vcpu="0" cpuset="0"/> <vcpupin vcpu="1" cpuset="4"/> <vcpupin vcpu="2" cpuset="1"/> <vcpupin vcpu="3" cpuset="5"/> <vcpupin vcpu="4" cpuset="2"/> <vcpupin vcpu="5" cpuset="6"/> <vcpupin vcpu="6" cpuset="3"/> <vcpupin vcpu="7" cpuset="7"/> </cputune> What I want is that VM1, VM2 and VM3 all receive half of the cores CPU time if congestion appears. My question is how the shares are calculated. Are they calculated on a per core basis? Meaning VM1 and VM3 use the first core so each will get half of the cores capacity when congestion appears, or are the shares calculated over all cores a virtual machine uses and I need to scale the amount of shares with increasing number of cores. What is the correct setting here so that all virtual machines in this example are guaranteed half of the capacity of each of their respective cores if congestion appears? Thank you.

2 1

Cron jobs not working with virsh commands
by Jeff Brown 26 May '23

26 May '23

Recently, while testing backup scripts, I happened to notice that 'virsh snaphot-create-as . . .' has been failing with 'error: failed to get domain '$DOMAIN'' when run through crontab. So I've been unwittingly backing up the live, running, QCOW2 images for quite some time because scripts which run fine when testing from CLI, fail to work as expected when run as a cron job. After much searching, I stumbled upon a related issue with the (obvious?) simple solution almost hidden in the answers. You have to explicitly connect to the local server with, e.g.: virsh -c qemu:///system snapshot-create-as . . . Humble apologies if I missed something I should have been aware of, but I just thought it worthwhile mentioning here for anyone else searching for an answer in the archives. Best regards, Jeff

2 1

fails to start @boot - service-start-limit-hit
by lejeczek 24 May '23

24 May '23

Hi guys. When I login to the system I find: UNIT LOAD ACTIVE SUB DESCRIPTION ● virtproxyd.service loaded failed failed Virtualization daemon ● virtproxyd-admin.socket loaded failed failed Libvirt proxy admin socket ● virtproxyd-ro.socket loaded failed failed Libvirt proxy local read-only socket ● virtproxyd-tls.socket loaded failed failed Libvirt proxy TLS IP socket ● virtproxyd.socket loaded failed failed Libvirt proxy local socket only 'virtproxyd-tls.socket' is enabled -> $ journalctl -l -o cat -u virtproxyd-tls.socket virtproxyd-tls.socket: Failed with result 'service-start-limit-hit'. Closed Libvirt proxy TLS IP socket. But any use of 'virtproxyd-tls.socket' from that point on, is successful - VM migrates to/from okey. I think pcs/pacemaker cluster plays a role in it for it must be - I'm guessing - trying to migrate a single VM back to booting-up node, though.. 'resource-agents-deps.target' is told to 'After=virtproxyd-tls.socket', among other 'after' bits. To confirm this suspicion - I put the node into 'standby' then I reboot and I get none of those failures mentioned above. & to make it an utterly curious test case - all these 'failures' only started after the node was remove from IPA domain ! If I make the node again a client/member of IPA (same as before) domain then - standby or not - all those services above fail @boot no more! I'd be very glad to read suggestions/thoughts you can share. many thanks, L.

1 0

Adding custom xml:ns to definitions for additional application
by Darragh Bailey 19 May '23

19 May '23

Hi, I'm looking at adding support to vagrant-libvirt to add some additional XML to the network definitions it creates following https://libvirt.org/formatnetwork.html#general-metadata assuming the metadata element is to be treated the same as what is listed for domains https://libvirt.org/formatdomain.html#general-metadata (app metadata isn't explained What I was wondering is if there are any expectations around the URI used? Or recommendations? I was hoping to use http://vagrant-libvirt.github.io/v1/network as it's at least under control of the maintainers. However I wasn't planning to publish anything there unless that is needed? Reason behind this is that I want to annotate networks created and managed by vagrant-libvirt so that teardown can be safely done without accidentally destroying networks that have been manually created. As well as allowing CI systems that might have multiple agents running on the same host to be able to safely create and destroy machines with network definitions in parallel with any attempts to remove a network that has just been created being prevented without networks ending up persisting forever. https://github.com/vagrant-libvirt/vagrant-libvirt/issues/1738 summaries what I'm hoping to do. -- Darragh Bailey "Nothing is foolproof to a sufficiently talented fool"

1 0

Invalid value '-1' for 'cpu.max': Invalid argument - a result of?
by lejeczek 17 May '23

17 May '23

Hi guys. In hope that an expert read this - what is, can be, the below a result of? 12284 still running (86040) Invalid value '-1' for 'cpu.max': Invalid argument 12284 still running (86035) 12284 still running (86030) this is a snippet from libvirtd logs which is a consequence of what ovirt's engine setup is doing. To troubleshoot ansible playbooks which is what engine setup does, as I understand it, would be an impossible task for me so I reckoned I should try this end. many thanks, L.

2 5