On 3/3/19 4:23 AM, Zhan Adlun wrote: This sounds like a deadlock. If you can attach gdb to libvirtd once virsh hangs and run 't a a bt' and share the output, that'd be great. Michal

Actually,I use the backdoors as you have mentioned. I use 'virsh qemu-monitor-command' to submit backup jobs and use 'virsh qemu-monitor-event DOMAIN --timestamp --loop' to monitor the jobs through the QMP protocol. for example : to do a full backup virsh qemu-monitor-command DOMAIN --pretty { "execute": "transaction", "arguments": { "actions": [ {"type": "block-dirty-bitmap-add", "data": {"node": "drive0", "name": "bitmap0"} }, {"type": "drive-backup", "data": {"device": "drive0", "target": "/path/to/full_backup.img", "sync": "full", "format": "qcow2"} } ] } } refer to https://wiki.qemu.org/Features/IncrementalBackup Unfortunately, the minitor process 'virsh qemu-monitor-event DOMAIN --timestamp --loop' was failed to be killed. A few hours later ,there were hundreds of monitor processes running(about 200-300), whice caused the 'virsh list' hung. After I have killed the processes, 'virsh list' works well. ________________________________ zhanhouliang(a)outlook.com 发件人： Eric Blake<mailto:eblake@redhat.com> 发送时间： 2019-03-06 23:43 收件人： Zhan Adlun<mailto:zhanhouliang@outlook.com>; libvirt-users@redhat.com<mailto:libvirt-users@redhat.com> 主题： Re: [libvirt-users] virsh hangs when backup jobs run on the kvm host On 3/2/19 9:23 PM, Zhan Adlun wrote: How are you using dirty bitmaps? As my new virDomainBackupBegin() API has not landed yet, I suspect you are using virQemuMonitorCommand() (virsh qemu-monitor-command) backdoors, which is explicitly unsupported (that is, if it breaks, you get to debug it - it exists for development work, and while 'query-*' QMP commands should be safe, commands that alter qemu state such as kicking off a backup job may break libvirt expectations), and which is WHY I'm trying to get incremental backup API officially supported. A precise sequence of commands that you issued in order to reproduce the virsh hang may be useful in debugging this. Hmm, my incremental backup API will require qemu 4.0; there have been a number of fixes in the qemu bitmap tracking that have landed since qemu 2.7.1. It could also be that you've hit an actual qemu bug where the QMP monitor has gone unresponsive. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org