significant delay live migrating a vm with an sr-iov interface
by Paul B. Henson
I'm using libvirt under Debian 12 (9.0.0-4+deb12u2 w/qemu
7.2+dfsg-7+deb12u12).
I have a vm using sr-iov, and configured it with a failover macvtap
interface so I could live migrate it. However, there is a significant
delay at the end of the migration resulting in a lot of lost traffic. If
I only have the macvtap interface, migration completes immediately at
the end of the transfer of memory with no loss of traffic.
I enabled debug logging, and found the following. On the source system,
it logs that the system is paused for the cutover:
2025-04-30 01:08:12.526+0000: 1696180: debug :
qemuMigrationAnyCompleted:1957 : Migration paused before switchover
at that point, for almost a minute, the source system just keeps
printing the same statistics:
2025-04-30 01:08:12.923+0000: 1696272: info :
qemuMonitorJSONIOProcessLine:208 : QEMU_MONITOR_RECV_REPLY:
mon=0x7f8fdc0ad2f0 reply={"return": {"expected-downtime": 300, "status":
"device", "setup-time": 297, "total-time": 26107, "ram": {"total":
137452265472, "postcopy-requests": 0, "dirty-sync-count": 3,
"multifd-bytes": 2821784576, "pages-per-second": 297855,
"downtime-bytes": 13208, "page-size": 4096, "remaining": 0,
"postcopy-bytes": 0, "mbps": 9786.9158461538464, "transferred":
3117658825, "dirty-sync-missed-zero-copy": 0, "precopy-bytes":
295861041, "duplicate": 32874480, "dirty-pages-rate": 56, "skipped": 0,
"normal-bytes": 2804301824, "normal": 684644}}, "id": "libvirt-577"}
[...]
2025-04-30 01:09:06.290+0000: 1696272: info :
qemuMonitorJSONIOProcessLine:208 : QEMU_MONITOR_RECV_REPLY:
mon=0x7f8fdc0ad2f0 reply={"return": {"expected-downtime": 300, "status":
"device", "setup-time": 297, "total-time"
: 79474, "ram": {"total": 137452265472, "postcopy-requests": 0,
"dirty-sync-count": 3, "multifd-bytes": 2821784576, "pages-per-second":
297855, "downtime-bytes": 13208, "page-size": 4096, "remaining": 0,
"postcopy-bytes"
: 0, "mbps": 9786.9158461538464, "transferred": 3117658825,
"dirty-sync-missed-zero-copy": 0, "precopy-bytes": 295861041,
"duplicate": 32874480, "dirty-pages-rate": 56, "skipped": 0,
"normal-bytes": 2804301824, "normal":
684644}}, "id": "libvirt-629"}
until finally it completes:
2025-04-30 01:09:06.327+0000: 1696272: info :
qemuMonitorJSONIOProcessLine:203 : QEMU_MONITOR_RECV_EVENT:
mon=0x7f8fdc0ad2f0 event={"timestamp": {"seconds": 1745975346,
"microseconds": 327382}, "event": "MIGRATION", "dat
a": {"status": "completed"}}
On the destination side, it says something about negotiating failover
for the network link:
2025-04-30 01:08:12.923+0000: 1384503: info :
qemuMonitorJSONIOProcessLine:203 : QEMU_MONITOR_RECV_EVENT: mon=
0x7fc7900ab2f0 event={"timestamp": {"seconds": 1745975292,
"microseconds": 922783}, "event": "FAILOVER_NEGOTIA
TED", "data": {"device-id": "ua-sr-iov-backup"}}
Then nothing happens for about a minute until it says it is done:
2025-04-30 01:09:06.328+0000: 1384503: debug :
qemuMonitorJSONIOProcessLine:189 : Line [{"timestamp": {"second
s": 1745975346, "microseconds": 327991}, "event": "MIGRATION", "data":
{"status": "completed"}}]
Any thoughts on what is going on here to cause this delay? It's clearly
somehow related to the sv-iov component of the migration.
Thanks much…
1 month, 2 weeks
virt-install iscsi direct - Target not found
by kgore4 une
When I try to use an iscsi direct pool in a "--disk" clause for
virt-install, I get error "iSCSI: Failed to connect to LUN : Failed to log
in to target. Status: Target not found(515)". I've seen that sort of error
before when the initiator name isn't used. The SAN returns different LUNS
depending on the initiator.
I've run out of ideas on what to try next. Any advice welcome. I've
included what I thought was relevent below.
klint.
The disk parameter to virt-install is (it's part of a script but the
variables are correct when executed)
[code]
--disk
vol=${poolName}/unit:0:0:${vLun1},xpath.set="./source/initiator/iqn/@name='iqn.2024-11.localdomain.agbu.agbuvh1:${vName}'"
\
[/code]
I added the xpath.set as I noticed that the initiator wasn't in the debug
output of the disk definition and it didn't work without it either.
The iscsi direct pool is defined and it appears to work - it's active and
vol-list shows the correct luns.
Using --debug on virt install, I can see the drive is detected early in the
process as it's got the size of the drive.
[code]
[Wed, 19 Feb 2025 16:43:33 virt-install 2174805] DEBUG (cli:3554) Parsed
--disk volume as: pool=agbu-ldap1 vol=unit:0:0:3
[Wed, 19 Feb 2025 16:43:33 virt-install 2174805] DEBUG (disk:648)
disk.set_vol_object: volxml=
<volume type='network'>
<name>unit:0:0:3</name>
<key>ip-10.1.4.3:3260-iscsi-iqn.1992-09.com.seagate:01.array.00c0fff6c846-lun-3</key>
<capacity unit='bytes'>49996103168</capacity>
<allocation unit='bytes'>49996103168</allocation>
<target>
<path>ip-10.1.4.3:3260-iscsi-iqn.1992-09.com.seagate:01.array.00c0fff6c846-lun-3</path>
</target>
</volume>
[Wed, 19 Feb 2025 16:43:33 virt-install 2174805] DEBUG (disk:650)
disk.set_vol_object: poolxml=
<pool type='iscsi-direct'>
<name>agbu-ldap1</name>
<uuid>1c4ae810-9bae-433c-a92f-7d3501b6ba80</uuid>
<capacity unit='bytes'>49996103168</capacity>
<allocation unit='bytes'>49996103168</allocation>
<available unit='bytes'>0</available>
<source>
<host name='10.1.4.3'/>
<device path='iqn.1992-09.com.seagate:01.array.00c0fff6c846'/>
<initiator>
<iqn name='iqn.2024-11.localdomain.agbu.agbuvh1:agbu-ldap'/>
</initiator>
</source>
</pool>
[/code]
The generated initial_xml for the disk looks like
[code]
<disk type="network" device="disk">
<driver name="qemu" type="raw"/>
<source protocol="iscsi"
name="iqn.1992-09.com.seagate:01.array.00c0fff6c846-lun-3">
<host name="10.1.4.3"/>
<initiator>
<iqn name="iqn.2024-11.localdomain.agbu.agbuvh1:agbu-ldap"/>
</initiator>
</source>
<target dev="vda" bus="virtio"/>
</disk>
[/code]
The generated final_xml looks like
[code]
<disk type="network" device="disk">
<driver name="qemu" type="raw"/>
<source protocol="iscsi"
name="iqn.1992-09.com.seagate:01.array.00c0fff6c846-lun-3">
<host name="10.1.4.3"/>
<initiator>
<iqn name="iqn.2024-11.localdomain.agbu.agbuvh1:agbu-ldap"/>
</initiator>
</source>
<target dev="vda" bus="virtio"/>
</disk>
[/code]
The full error is
[code]
[Wed, 19 Feb 2025 16:43:35 virt-install 2174805] DEBUG (cli:256) File
"/usr/bin/virt-install", line 8, in <module>
virtinstall.runcli()
File "/usr/share/virt-manager/virtinst/virtinstall.py", line 1233, in
runcli
sys.exit(main())
File "/usr/share/virt-manager/virtinst/virtinstall.py", line 1226, in main
start_install(guest, installer, options)
File "/usr/share/virt-manager/virtinst/virtinstall.py", line 974, in
start_install
fail(e, do_exit=False)
File "/usr/share/virt-manager/virtinst/cli.py", line 256, in fail
log.debug("".join(traceback.format_stack()))
[Wed, 19 Feb 2025 16:43:35 virt-install 2174805] ERROR (cli:257) internal
error: process exited while connecting to monitor:
2025-02-19T05:43:35.075695Z qemu-system-x86_64: -blockdev
{"driver":"iscsi","portal":"10.1.4.3:3260","target":"iqn.1992-09.com.seagate:01.array.00c0fff6c846-lun-3","lun":0,"transport":"tcp","initiator-name":"iqn.2024-11.localdomain.agbu.agbuvh1:agbu-ldap","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}:
iSCSI: Failed to connect to LUN : Failed to log in to target. Status:
Target not found(515)
[Wed, 19 Feb 2025 16:43:35 virt-install 2174805] DEBUG (cli:259)
Traceback (most recent call last):
File "/usr/share/virt-manager/virtinst/virtinstall.py", line 954, in
start_install
domain = installer.start_install(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/virt-manager/virtinst/install/installer.py", line 695,
in start_install
domain = self._create_guest(
^^^^^^^^^^^^^^^^^^^
File "/usr/share/virt-manager/virtinst/install/installer.py", line 637,
in _create_guest
domain = self.conn.createXML(initial_xml or final_xml, 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/libvirt.py", line 4481, in createXML
raise libvirtError('virDomainCreateXML() failed')
libvirt.libvirtError: internal error: process exited while connecting to
monitor: 2025-02-19T05:43:35.075695Z qemu-system-x86_64: -blockdev
{"driver":"iscsi","portal":"10.1.4.3:3260","target":"iqn.1992-09.com.seagate:01.array.00c0fff6c846-lun-3","lun":0,"transport":"tcp","initiator-name":"iqn.2024-11.localdomain.agbu.agbuvh1:agbu-ldap","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}:
iSCSI: Failed to connect to LUN : Failed to log in to target. Status:
Target not found(515)
[/code]
Things that could affect the answer
* What I'm calling a SAN is a seagate exos-x iscsi unit
* virtual host is debian 12
* virsh version 9.0.0
* iscsiadm version 2.1.8
1 month, 2 weeks
Guest cannot access host HTTP service in NAT
by icefrog1950＠gmail.com
Hi,
I hit a libvirt networking problem that guest cannot access host HTTP service. I debug this issue and tried some efforts. Thanks for your suggestions!
Environment
-------------
guest IP: 192.168.122.46 (Linux, default NAT, installed using virt-manager)
host1 IP: 192.168.3.16 (Centos 8.5 running libvirt and qemu, default libvirt iptable rules)
HTTP service: 192.168.3.16:70 (firewall rules have allowed this port)
host2: 192.168.3.65:70 (for test only)
guest network xml
<interface type="network">
<mac address="52:54:00:f5:a8:9f"/>
<source network="default" portid="6e8ce7e7-6517-43fa-b113-aaddb6c1bc08" bridge="virbr0"/>
<target dev="vnet2"/>
<model type="e1000"/>
<alias name="net0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
</interface>
1. guest and host1/host2 can ping each other
2. *guest can visit host2 HTTP
3. *guest cannot visit host1 HTTP
When I capture traffic in guest, Wireshark shows:
192.168.122.46 -> 192.168.3.16 // SYN ok
192.168.122.46 <- 192.168.3.16 // Destination unreachable (Port unreachable)
guest route table:
------------------
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.122.1 0.0.0.0 UG 100 0 0 eth0
192.168.122.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
host1 route table:
------------------
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.3.1 0.0.0.0 UG 100 0 0 enp3s0
192.168.3.0 0.0.0.0 255.255.255.0 U 100 0 0 enp3s0
192.168.122.0 0.0.0.0 255.255.255.0 UG 0 0 0 virbr0
I delete the last rule and add a rule to make sure host1 visits guests will go through 192.168.122.1
------------------
Destination Gateway Genmask Flags Metric Ref Use Iface
(other rules omitted)
192.168.122.0 192.168.122.1 255.255.255.0 UG 0 0 0 virbr0
However, traceroute shows this new rule does not work (which should go to 192.168.122.1 first), and guest cannot visit host1 HTTP request.
[!] guest visit http://192.168.3.16:70 does not go through 192.168.122.1
traceroute 192.168.122.46
traceroute to 192.168.122.46 (192.168.122.46), 30 hops max, 60 byte packets
1 192.168.122.46 (192.168.122.46) 0.200 ms 0.194 ms 0.194 ms
# guest visit http://192.168.3.65:70 goes through 192.168.122.1
traceroute 192.168.3.65
traceroute to 192.168.3.65 (192.168.3.65), 30 hops max, 60 byte packets
1 192.168.122.1 (192.168.122.1) 0.244 ms 0.050 ms 0.120 ms
2 192.168.3.65 (192.168.3.65) 0.823 ms !X 0.802 ms !X 0.789 ms !X
In sum, is there a way to force the guest go through 192.168.122.1 when visiting the hosting machine?
-------------------
libvirt iptable rules:
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N LIBVIRT_INP
-N LIBVIRT_OUT
-N LIBVIRT_FWO
-N LIBVIRT_FWI
-N LIBVIRT_FWX
-A INPUT -j LIBVIRT_INP
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -j LIBVIRT_OUT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT
-A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable # delete this rule does not work
-A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable # delete this rule does not work
-A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT
1 month, 3 weeks
vm stopped booting
by daggs
Greetings,
my windows vm stoop working, I think it is related kernel version, 6.14.1 works but 6.14.2 seems not to
looking at the libvirt-qemu log file shows this:
2025-04-22 17:28:52.082+0000: starting up libvirt version: 11.2.0, qemu version: 9.2.3, kernel: 6.14.3-gentoo, hostname: NCC-5001D.lan
LC_ALL=C \
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin:/usr/lib/llvm/19/bin:/etc/eselect/wine/bin:/opt/cuda/bin \
HOME=/var/lib/libvirt/qemu/domain-1-windows \
USER=root \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-windows/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-windows/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-windows/.config \
/usr/bin/qemu-system-x86_64 \
-name guest=windows,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-windows/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/edk2/OvmfX64/OVMF_CODE_4M.secboot.qcow2","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"qcow2","file":"libvirt-pflash0-storage","backing":null}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/windows_VARS.qcow2","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"qcow2","file":"libvirt-pflash1-storage","backing":null}' \
-machine pc-q35-8.1,usb=off,smm=on,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,hpet=off,acpi=on \
-accel kvm \
-cpu host,migratable=on,topoext=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=wateverr,kvm=off,host-cache-info=on,l3-cache=off \
-global driver=cfi.pflash01,property=secure,value=on \
-m size=31457280k \
-overcommit mem-lock=off \
-smp 16,sockets=1,dies=1,clusters=1,cores=8,threads=2 \
-object '{"qom-type":"memory-backend-memfd","id":"ram-node0","share":true,"size":32212254720}' \
-numa node,nodeid=0,cpus=0-15,memdev=ram-node0 \
-uuid e8fc4f01-5b87-4e2d-a30f-fb72318688f7 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=27,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \
-device '{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}' \
-device '{"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"}' \
-device '{"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"}' \
-device '{"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"}' \
-device '{"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"}' \
-device '{"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"}' \
-device '{"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"}' \
-device '{"driver":"virtio-scsi-pci","id":"scsi0","bus":"pcie.0","addr":"0xc"}' \
-blockdev '{"driver":"file","filename":"/usr/share/drivers/windows/virtio-win.iso","node-name":"libvirt-1-storage","read-only":true}' \
-device '{"driver":"ide-cd","bus":"ide.1","drive":"libvirt-1-storage","id":"sata0-0-1"}' \
-chardev socket,id=chr-vu-fs0,path=/var/lib/libvirt/qemu/domain-1-windows/fs0-fs.sock \
-device '{"driver":"vhost-user-fs-pci","id":"fs0","chardev":"chr-vu-fs0","queue-size":1024,"tag":"home@linux","bus":"pci.6","addr":"0x0"}' \
-netdev '{"type":"tap","fd":"29","vhost":true,"vhostfd":"31","id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:43:79:55","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/1-windows-swtpm.sock \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-object '{"qom-type":"input-linux","id":"input0","evdev":"/var/lib/libvirt/helpers_state/windows/mouse-event"}' \
-object '{"qom-type":"input-linux","id":"input1","evdev":"/var/lib/libvirt/helpers_state/windows/keyboard-event","repeat":true,"grab_all":true,"grab-toggle":"ctrl-ctrl"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-global ICH9-LPC.noreboot=off \
-watchdog-action reset \
-device '{"driver":"vfio-pci","host":"0000:06:00.0","id":"hostdev0","bus":"pci.5","multifunction":true,"addr":"0x0","romfile":"/var/lib/libvirt/roms/gpu-10de:1c82-uefi.rom"}' \
-device '{"driver":"vfio-pci","host":"0000:06:00.1","id":"hostdev1","bus":"pci.5","addr":"0x0.0x1"}' \
-device '{"driver":"vfio-pci","host":"0000:04:00.1","id":"hostdev2","bus":"pci.7","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:08:00.4","id":"hostdev3","bus":"pci.4","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev4","bus":"pci.9","addr":"0x0"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/003/002","id":"hostdev5","bus":"usb.0","port":"1"}' \
-device '{"driver":"usb-host","id":"hostdev6","bus":"usb.0","port":"2"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/002","id":"hostdev7","bus":"usb.0","port":"3"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.3","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2025-04-22 17:28:52.084+0000: 5390: debug : virProcessSetMaxMemLock:923 : Locked memory for process 5390 limited to 33285996544 bytes
2025-04-22 17:28:52.084+0000: 5390: debug : virExec:881 : Run hook 0x7fb3b82902a0 0x7fb3bb4435d0
2025-04-22 17:28:52.084+0000: 5390: debug : qemuProcessHook:3136 : Obtaining domain lock
2025-04-22 17:28:52.084+0000: 5390: debug : virDomainLockProcessStart:175 : plugin=0x7fb37016de00 dom=0x7fb3704b9bd0 paused=1 fd=0x7fb3bb4432f0
2025-04-22 17:28:52.084+0000: 5390: debug : virDomainLockManagerNew:130 : plugin=0x7fb37016de00 dom=0x7fb3704b9bd0 withResources=1
2025-04-22 17:28:52.084+0000: 5390: debug : virLockManagerPluginGetDriver:275 : plugin=0x7fb37016de00
2025-04-22 17:28:52.084+0000: 5390: debug : virLockManagerNew:298 : driver=0x7fb3c09e5440 type=0 nparams=5 params=0x7fb3bb4431d0 flags=0x1
2025-04-22 17:28:52.084+0000: 5390: debug : virLockManagerLogParams:96 : key=uuid type=uuid value=e8fc4f01-5b87-4e2d-a30f-fb72318688f7
2025-04-22 17:28:52.084+0000: 5390: debug : virLockManagerLogParams:89 : key=name type=string value=windows
2025-04-22 17:28:52.084+0000: 5390: debug : virLockManagerLogParams:77 : key=id type=uint value=1
2025-04-22 17:28:52.084+0000: 5390: debug : virLockManagerLogParams:77 : key=pid type=uint value=5390
2025-04-22 17:28:52.084+0000: 5390: debug : virLockManagerLogParams:92 : key=uri type=cstring value=qemu:///system
2025-04-22 17:28:52.084+0000: 5390: debug : virDomainLockManagerNew:143 : Adding leases
2025-04-22 17:28:52.084+0000: 5390: debug : virDomainLockManagerNew:148 : Adding disks
2025-04-22 17:28:52.084+0000: 5390: debug : virDomainLockManagerAddImage:87 : Add disk /usr/share/drivers/windows/virtio-win.iso
2025-04-22 17:28:52.084+0000: 5390: debug : virLockManagerAddResource:324 : lock=0x7fb3b0044350 type=0 name=/usr/share/drivers/windows/virtio-win.iso nparams=0 params=(nil) flags=0x1
2025-04-22 17:28:52.084+0000: 5390: debug : virLockManagerAcquire:342 : lock=0x7fb3b0044350 state='<null>' flags=0x3 action=0 fd=0x7fb3bb4432f0
2025-04-22 17:28:52.084+0000: 5390: debug : virLockManagerFree:380 : lock=0x7fb3b0044350
2025-04-22 17:28:52.084+0000: 5390: info : virObjectUnref:378 : OBJECT_UNREF: obj=0x7fb370114eb0
2025-04-22 17:28:52.084+0000: 5390: debug : qemuProcessHook:3181 : Hook complete ret=0
2025-04-22 17:28:52.084+0000: 5390: debug : virExec:883 : Done hook 0
2025-04-22 17:28:52.084+0000: 5390: debug : virExecCommon:456 : Setting child uid:gid to 77:77 with caps 0
2025-04-22 17:28:52.084+0000: 5390: debug : virCommandHandshakeChild:402 : Notifying parent for handshake start on 33
2025-04-22 17:28:52.084+0000: 5390: debug : virCommandHandshakeChild:410 : Waiting on parent for handshake complete on 34
2025-04-22 17:28:52.100+0000: 5390: error : virCommandHandshakeChild:418 : libvirtd quit during handshake: Input/output error
libvirt: error : libvirtd quit during handshake: Input/output error
2025-04-22 17:28:52.100+0000: shutting down, reason=failed
any hints on what can cause the handshake error?
3 months, 1 week
virt-manager with SSH and 2FA with TOTP?
by Andreas Haumer
Hi!
(I hope, this is the right list for my question. I already
posted it to the debian-user ML, but someone pointed me to
this list. Alas, there is no virt-manager ML anymore)
In our network we have several Debian systems working as VM host
running QEMU+KVM based virtual machines.
I usually use virt-manager on my workstation as GUI to connect
to the VM host, manage the VMs and also to connect to the VM
console if needed.
To connect to the VM host I use SSH with public key authentication.
On the commandline with virsh this looks like this (example):
andreas@ws1:~> virsh -c qemu+ssh://root@maxwell/system
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh #
So far, so good.
Recently I decided to increase our internal network security standards
and activated 2FA with time-based one-time passwords on several hosts.
(The idea is to eventually have 2FA for SSH for all users on all hosts
in our network)
This works very well and even quite comfortable with authenticator-apps
on my smartphone or KeePassXC on my workstation generating the TOTP.
Example:
andreas@ws1:~> ssh root@mach
Enter OTP:
Linux mach 6.1.0-32-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.129-1 (2025-03-06) x86_64
root@mach:~#
So for a successful SSH connection I now have to enter a valid TOTP (generated by the
authenticator app) and then it connects.
Connecting to the host with virsh on the commandline also works in a similar way:
andreas@ws1:~> virsh -c qemu+ssh://root@mach/system
Enter OTP:
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh #
All fine. Works as designed...
When I use virt-manager to connect to the VM host, the GUI opens
a dialog asking for the OTP and then connects, showing the list of
all configured VMs etc. I can also open the configuration of a
given VM, manage and change it.
All fine, too...
But when I try to use virt-manager to connect to the console of a
specific VM, it doesn't work as expected.
virt-manager opens a new window for the console, but also endlessly
keeps opening password entry dialogs.
As soon as I enter the current OTP and klick "ok", another dialog
is opened, again asking for another OTP. And so on...
(These are one-time passwords, valid for 30 seconds, which cannot be re-used)
I can connect to the VM console with a SPICE viewer like remmina
using SSH port forwarding like this:
andreas@ws1:~> ssh -L 5906:localhost:5906 root@mach
Enter OTP:
root@mach:~#
(where 5906 is the SPICE port for the VM in question)
And then use remmina to connect to port 5906 on localhost.
This gives me the SPICE console of the VM.
Of course, this is not as comfortable as using virt-manager.
But with virt-manager I haven't found a way to successfully
connect to the VM console with 2FA in place.
So, finally, my question: Did anyone on this list manage to
use virt-manager to connect to a VM console using SSH with 2FA?
Thanks!
- andreas
--
Andreas Haumer
*x Software + Systeme | mailto:andreas@xss.co.at
Karmarschgasse 51/2/20 | https://www.xss.co.at/
A-1100 Vienna, Austria | Tel: +43-1-6060114
3 months, 2 weeks
Is there any way to apply firewall rules to passt user mode
networking?
by notsyncing＠163.com
Hello, I'm testing user mode network under session mode with libvirt and passt backend, and I found that the guest vm can access the whole host LAN(host is 192.168.1.2, guest ip configured to 192.168.100.2, but guest can access 192.168.1.3, which should not be allowed). Using firewalld dropping all traffic from guest ip has no effect at all. Is there any way to prevent guest accessing host LAN, while keeping guest's internet access? Thanks!
3 months, 2 weeks
iptables rule affects vnet0?
by daggs
Greetings,
I have a 2 vms on a host that can communicate with other hosts on the system.
the two vms are connected by a virtsw0 bridge (vm1 and vm2) and on of the vms (vm1) has another connection (vnet0) to the host.
it seems that the host can only communicate with the vm that has a direct connection to it and not the other vm (virtsw0 allows connection beteen both vms)
here are the rules libvirt creates:
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N LIBVIRT_FWI
-N LIBVIRT_FWO
-N LIBVIRT_FWX
-N LIBVIRT_INP
-N LIBVIRT_OUT
-A INPUT -j LIBVIRT_INP
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A OUTPUT -j LIBVIRT_OUT
-A LIBVIRT_FWI -o virsw0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -i virsw0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX -i virsw0 -o virsw0 -j ACCEPT
-A LIBVIRT_INP -i virsw0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virsw0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virsw0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virsw0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virsw0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virsw0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virsw0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virsw0 -p tcp -m tcp --dport 68 -j ACCEPT
and here are the stats:
Chain INPUT (policy ACCEPT 17405 packets, 1677K bytes)
pkts bytes target prot opt in out source destination
17434 1688K LIBVIRT_INP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 LIBVIRT_FWX all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LIBVIRT_FWI all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 LIBVIRT_FWO all -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 21058 packets, 3788K bytes)
pkts bytes target prot opt in out source destination
21058 3788K LIBVIRT_OUT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain LIBVIRT_FWI (1 references)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- * virsw0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain LIBVIRT_FWO (1 references)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- virsw0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain LIBVIRT_FWX (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- virsw0 virsw0 0.0.0.0/0 0.0.0.0/0
Chain LIBVIRT_INP (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- virsw0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- virsw0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
29 10608 ACCEPT udp -- virsw0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67
0 0 ACCEPT tcp -- virsw0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
Chain LIBVIRT_OUT (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- * virsw0 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- * virsw0 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0 0 ACCEPT udp -- * virsw0 0.0.0.0/0 0.0.0.0/0 udp dpt:68
0 0 ACCEPT tcp -- * virsw0 0.0.0.0/0 0.0.0.0/0 tcp dpt:68
is it possible I cannot access vm2 from the host because of the rules above?
Thanks,
Dagg
3 months, 2 weeks
How to use microVM as a machine type?
by Qu Wenruo
Hi,
Recently I'm hitting a bug related to edk2 that sometimes boot up can be
very slow (before loading the bootloader):
https://github.com/tianocore/edk2/issues/10765
It looks like the cause is the related to flash IOs.
So I want to try using microVM machine type, as it's json file describes
the device mapping is "memory", which should work around the slow boot.
But I'm unable to setup a machine using microVM machine type in
virtual-manager.
And manually editing the xml file doesn't let me go any further, as VMM
creates a default xml with ACPI feature, which is not supported by microVM.
So I'm wondering, what's the proper way to use microVM machine type
inside libvirt?
Thanks,
Qu
3 months, 3 weeks
