4:23 a.m.
I'm trying to create a hook that attaches several virtio-scsi disks to a starting vm,
and it seems like the hook enters a deadlock while attempting to do so.
Is there any workaround around this? Any better way, forking virsh execution somehow?
Could anyone explain why this is creating a deadlock in the first place? Isn't libvirt
multithreaded?
Thanks for the help!
Shlomi
### hook script:
#!/bin/bash
enum_scsi() {
scsi_devices=`cd /dev;ls sd[^a]*`
}
gen_hba_xml() {
cat > ./hba.xml << EOF
<controller type='scsi' model='virtio-scsi'/>
EOF
}
passthrough_start() {
# create virtio-scsi HBA
gen_hba_xml
virsh attach-device --persistent $domain_name ./hba.xml
rm -f ./hba.xml
# attache scsi devices to guest
enum_scsi
for sd in ${scsi_devices[@]}
do
virsh attach-disk --persistent $domain_name /dev/${sd} ${sd}
done
}
domain_name=$1
domain_event=$2
case $domain_event in
start)
passthrough_start
;;
*)
echo "qemu hook called with unexpected options $*" >&2
;;
esac
### starce libvirt:
[root@localhost ~]# strace -f -p 14630
Process 14630 attached with 11 threads - interrupt to quit
[pid 14640] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...>
[pid 14638] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...>
[pid 14637] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...>
[pid 14636] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...>
[pid 14635] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 14639] futex(0xc34ebc, FUTEX_WAIT_PRIVATE, 2303, NULL <unfinished ...>
[pid 14634] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 14631] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
[pid 14633] futex(0x7fcc240ccd80, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 14630] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
[pid 14632] futex(0xc34e24, FUTEX_WAIT_PRIVATE, 3091, NULL^C <unfinished ...>
...
[pid 14630] poll([{fd=7, events=POLLIN}, {fd=9, events=POLLIN}, {fd=12, events=POLLIN},
{fd=13, events=POLLIN}, {fd=6, events=POLLIN}, {fd=14, events=POLLIN}, {fd=16,
events=POLLIN}, {fd=18, events=POLLIN|POLLOUT}, {fd=24, events=POLLIN}], 9, 1927) = 1
([{fd=18, revents=POLLOUT}])
[pid 14630] gettid() = 14630
[pid 14630] gettid() = 14630
[pid 14630] gettid() = 14630
[pid 14630] gettid() = 14630
[pid 14630] gettid() = 14630
[pid 14630] gettid() = 14630
[pid 14630] gettid() = 14630
[pid 14630] gettid() = 14630
[pid 14630] gettid() = 14630
[pid 14630] gettid() = 14630
[pid 14630] gettid() = 14630
[pid 14630] gettid() = 14630
[pid 14630] write(18, "\0\0\0\34keep\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0",
28) = 28
...
### strace bash:
[root@localhost ~]# strace -f -p 14783
Process 14783 attached - interrupt to quit
wait4(-1,
...
9:17 a.m.
On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
I'm trying to create a hook that attaches several virtio-scsi
disks to a
starting vm,
and it seems like the hook enters a deadlock while attempting to do so.
Is there any workaround around this? Any better way, forking virsh
execution somehow?
Could anyone explain why this is creating a deadlock in the first place?
Isn't libvirt multithreaded?
Thanks for the help!
Shlomi
### hook script:
#!/bin/bash
enum_scsi() {
scsi_devices=`cd /dev;ls sd[^a]*`
}
gen_hba_xml() {
cat > ./hba.xml << EOF
<controller type='scsi' model='virtio-scsi'/>
EOF
}
passthrough_start() {
# create virtio-scsi HBA
gen_hba_xml
virsh attach-device --persistent $domain_name ./hba.xml
rm -f ./hba.xml
http://libvirt.org/hooks.html#recursive
Do not do this.
Michal
9:59 a.m.
Yes.. I saw that, but I have to :)
The issue was solved by converting the hook to Python and using
'subprocess.Popen'. deadlock is gone, all looks good.
Regards
Shlomi
-----Original Message-----
From: Michal Privoznik [mailto:mprivozn@redhat.com]
Sent: Wednesday, November 06, 2013 5:17 PM
To: TSADOK, Shlomi (Shlomi)
Cc: libvirt-users(a)redhat.com
Subject: Re: [libvirt-users] Calling 'virsh' from within a hook script - avoid
deadlock?
On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
I'm trying to create a hook that attaches several virtio-scsi
disks to
a starting vm,
and it seems like the hook enters a deadlock while attempting to do so.
Is there any workaround around this? Any better way, forking virsh
execution somehow?
Could anyone explain why this is creating a deadlock in the first place?
Isn't libvirt multithreaded?
Thanks for the help!
Shlomi
### hook script:
#!/bin/bash
enum_scsi() {
scsi_devices=`cd /dev;ls sd[^a]*`
}
gen_hba_xml() {
cat > ./hba.xml << EOF
<controller type='scsi' model='virtio-scsi'/>
EOF
}
passthrough_start() {
# create virtio-scsi HBA
gen_hba_xml
virsh attach-device --persistent $domain_name ./hba.xml
rm -f ./hba.xml
http://libvirt.org/hooks.html#recursive
Do not do this.
Michal
10:07 p.m.
On Wed, Nov 06, 2013 at 03:59:28PM +0000, TSADOK, Shlomi (Shlomi) wrote:
Yes.. I saw that, but I have to :)
The issue was solved by converting the hook to Python and using
'subprocess.Popen'. deadlock is gone, all looks good.
If that works it is pretty much by luck and not by design. We
make *zero* guarantees that you can call libvirt from a hook
function. Don't be surprised if your code breaks in the future
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
3:16 a.m.
On 06.11.2013 16:17, Michal Privoznik wrote:
On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
> [...]
http://libvirt.org/hooks.html#recursive
Do not do this.
Michal
Maybe I should explain a bit why this is not supported. It's true that
libvirt is multithreaded. However, not in terms of multiple write/modify
accesses to a single domain. That is, when a domain is being started up
it is a modify access. Meanwhile, at some point during the transaction
the startup hook script is called. If the script decides to start
another modify transaction an deadlock occurs.
Michal
3:19 a.m.
On Thu, Nov 07, 2013 at 10:16:29AM +0100, Michal Privoznik wrote:
On 06.11.2013 16:17, Michal Privoznik wrote:
> On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
>> [...]
>
> http://libvirt.org/hooks.html#recursive
>
> Do not do this.
>
> Michal
>
Maybe I should explain a bit why this is not supported. It's true that
libvirt is multithreaded. However, not in terms of multiple write/modify
accesses to a single domain. That is, when a domain is being started up
it is a modify access. Meanwhile, at some point during the transaction
the startup hook script is called. If the script decides to start
another modify transaction an deadlock occurs.
There is also a limit to the number of threads which deal with jobs,
so even if the threads are all busy, your hook will stall and potentially
even deadlock if all creates are stuck in create calls with hooks running
even across different VMs.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
5:11 a.m.
Thanks for the explanation(and warnings). Very interesting and useful feature.
Shlomi
-----Original Message-----
From: Daniel P. Berrange [mailto:berrange@redhat.com]
Sent: Thursday, November 07, 2013 11:20 AM
To: Michal Privoznik
Cc: TSADOK, Shlomi (Shlomi); libvirt-users(a)redhat.com
Subject: Re: [libvirt-users] Calling 'virsh' from within a hook script - avoid
deadlock?
On Thu, Nov 07, 2013 at 10:16:29AM +0100, Michal Privoznik wrote:
On 06.11.2013 16:17, Michal Privoznik wrote:
> On 06.11.2013 11:23, TSADOK, Shlomi (Shlomi) wrote:
>> [...]
>
> http://libvirt.org/hooks.html#recursive
>
> Do not do this.
>
> Michal
>
Maybe I should explain a bit why this is not supported. It's true that
libvirt is multithreaded. However, not in terms of multiple
write/modify accesses to a single domain. That is, when a domain is
being started up it is a modify access. Meanwhile, at some point
during the transaction the startup hook script is called. If the
script decides to start another modify transaction an deadlock occurs.
There is also a limit to the number of threads which deal with jobs, so even if the
threads are all busy, your hook will stall and potentially even deadlock if all creates
are stuck in create calls with hooks running even across different VMs.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
4033
days inactive
4034
days old
6 comments
3 participants
participants (3)
-
Daniel P. Berrange -
Michal Privoznik -
TSADOK, Shlomi (Shlomi)