[libvirt-users] Calling 'virsh' from within a hook script - avoid deadlock?
I'm trying to create a hook that attaches several virtio-scsi disks to a starting vm, and it seems like the hook enters a deadlock while attempting to do so. Is there any workaround around this? Any better way, forking virsh execution somehow? Could anyone explain why this is creating a deadlock in the first place? Isn't libvirt multithreaded? Thanks for the help! Shlomi ### hook script: #!/bin/bash enum_scsi() { scsi_devices=`cd /dev;ls sd[^a]*` } gen_hba_xml() { cat > ./hba.xml << EOF <controller type='scsi' model='virtio-scsi'/> EOF } passthrough_start() { # create virtio-scsi HBA gen_hba_xml virsh attach-device --persistent $domain_name ./hba.xml rm -f ./hba.xml # attache scsi devices to guest enum_scsi for sd in ${scsi_devices[@]} do virsh attach-disk --persistent $domain_name /dev/${sd} ${sd} done } domain_name=$1 domain_event=$2 case $domain_event in start) passthrough_start ;; *) echo "qemu hook called with unexpected options $*" >&2 ;; esac ### starce libvirt: [root@localhost ~]# strace -f -p 14630 Process 14630 attached with 11 threads - interrupt to quit [pid 14640] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...> [pid 14638] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...> [pid 14637] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...> [pid 14636] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...> [pid 14635] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> [pid 14639] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...> [pid 14634] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> [pid 14631] restart_syscall(<... resuming interrupted call ...> <unfinished ...> [pid 14633] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> [pid 14630] restart_syscall(<... resuming interrupted call ...> <unfinished ...> [pid 14632] futex(0xc34e24, FUTEX_WAIT_PRIVATE, 3091, NULL^C <unfinished ...> ... [pid 14630] poll([{fd=7, events=POLLIN}, {fd=9, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=6, events=POLLIN}, {fd=14, events=POLLIN}, {fd=16, events=POLLIN}, {fd=18, events=POLLIN|POLLOUT}, {fd=24, events=POLLIN}], 9, 1927) = 1 ([{fd=18, revents=POLLOUT}]) [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] gettid() = 14630 [pid 14630] write(18, "\0\0\0\34keep\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0", 28) = 28 ... ### strace bash: [root@localhost ~]# strace -f -p 14783 Process 14783 attached - interrupt to quit wait4(-1, ...
On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
I'm trying to create a hook that attaches several virtio-scsi disks to a starting vm,
and it seems like the hook enters a deadlock while attempting to do so.
Is there any workaround around this? Any better way, forking virsh execution somehow?
Could anyone explain why this is creating a deadlock in the first place? Isn't libvirt multithreaded?
Thanks for the help!
Shlomi
### hook script:
#!/bin/bash
enum_scsi() {
scsi_devices=`cd /dev;ls sd[^a]*`
}
gen_hba_xml() {
cat > ./hba.xml << EOF
<controller type='scsi' model='virtio-scsi'/>
EOF
}
passthrough_start() {
# create virtio-scsi HBA
gen_hba_xml
virsh attach-device --persistent $domain_name ./hba.xml
rm -f ./hba.xml
http://libvirt.org/hooks.html#recursive Do not do this. Michal
Yes.. I saw that, but I have to :) The issue was solved by converting the hook to Python and using 'subprocess.Popen'. deadlock is gone, all looks good. Regards Shlomi -----Original Message----- From: Michal Privoznik [mailto:mprivozn@redhat.com] Sent: Wednesday, November 06, 2013 5:17 PM To: TSADOK, Shlomi (Shlomi) Cc: libvirt-users@redhat.com Subject: Re: [libvirt-users] Calling 'virsh' from within a hook script - avoid deadlock? On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
I'm trying to create a hook that attaches several virtio-scsi disks to a starting vm,
and it seems like the hook enters a deadlock while attempting to do so.
Is there any workaround around this? Any better way, forking virsh execution somehow?
Could anyone explain why this is creating a deadlock in the first place? Isn't libvirt multithreaded?
Thanks for the help!
Shlomi
### hook script:
#!/bin/bash
enum_scsi() {
scsi_devices=`cd /dev;ls sd[^a]*`
}
gen_hba_xml() {
cat > ./hba.xml << EOF
<controller type='scsi' model='virtio-scsi'/>
EOF
}
passthrough_start() {
# create virtio-scsi HBA
gen_hba_xml
virsh attach-device --persistent $domain_name ./hba.xml
rm -f ./hba.xml
http://libvirt.org/hooks.html#recursive Do not do this. Michal
On Wed, Nov 06, 2013 at 03:59:28PM +0000, TSADOK, Shlomi (Shlomi) wrote:
Yes.. I saw that, but I have to :)
The issue was solved by converting the hook to Python and using 'subprocess.Popen'. deadlock is gone, all looks good.
If that works it is pretty much by luck and not by design. We make *zero* guarantees that you can call libvirt from a hook function. Don't be surprised if your code breaks in the future Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 06.11.2013 16:17, Michal Privoznik wrote:
On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
[...]
http://libvirt.org/hooks.html#recursive
Do not do this.
Michal
Maybe I should explain a bit why this is not supported. It's true that libvirt is multithreaded. However, not in terms of multiple write/modify accesses to a single domain. That is, when a domain is being started up it is a modify access. Meanwhile, at some point during the transaction the startup hook script is called. If the script decides to start another modify transaction an deadlock occurs. Michal
On Thu, Nov 07, 2013 at 10:16:29AM +0100, Michal Privoznik wrote:
On 06.11.2013 16:17, Michal Privoznik wrote:
On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
[...]
http://libvirt.org/hooks.html#recursive
Do not do this.
Michal
Maybe I should explain a bit why this is not supported. It's true that libvirt is multithreaded. However, not in terms of multiple write/modify accesses to a single domain. That is, when a domain is being started up it is a modify access. Meanwhile, at some point during the transaction the startup hook script is called. If the script decides to start another modify transaction an deadlock occurs.
There is also a limit to the number of threads which deal with jobs, so even if the threads are all busy, your hook will stall and potentially even deadlock if all creates are stuck in create calls with hooks running even across different VMs. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Thanks for the explanation(and warnings). Very interesting and useful feature. Shlomi -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Thursday, November 07, 2013 11:20 AM To: Michal Privoznik Cc: TSADOK, Shlomi (Shlomi); libvirt-users@redhat.com Subject: Re: [libvirt-users] Calling 'virsh' from within a hook script - avoid deadlock? On Thu, Nov 07, 2013 at 10:16:29AM +0100, Michal Privoznik wrote:
On 06.11.2013 16:17, Michal Privoznik wrote:
On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
[...]
http://libvirt.org/hooks.html#recursive
Do not do this.
Michal
Maybe I should explain a bit why this is not supported. It's true that libvirt is multithreaded. However, not in terms of multiple write/modify accesses to a single domain. That is, when a domain is being started up it is a modify access. Meanwhile, at some point during the transaction the startup hook script is called. If the script decides to start another modify transaction an deadlock occurs.
There is also a limit to the number of threads which deal with jobs, so even if the threads are all busy, your hook will stall and potentially even deadlock if all creates are stuck in create calls with hooks running even across different VMs. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (3)
-
Daniel P. Berrange -
Michal Privoznik -
TSADOK, Shlomi (Shlomi)