Han, Did the same test again with libvirt debug enabled. This time I added the disk[1] without snaps (OK), and then added the disk[2] with snapshots. The libvirt log[3] (line 320) says: *Could not open '/opt/stack/data/nova/mnt/a3b4c6ddd9bf82edd4f726872be58d05/volume-a75b42e8-8175-4516-b724-43ee6f3d2520': Permission denied\r\n", "id": "libvirt-33"}]* But this file has 777 permissions. I can access it with any user I tried[4]. Tough its mounted over NFS, the permissions seems to be right. Erlon [1] http://paste.openstack.org/show/556055/ [2] http://paste.openstack.org/show/556056/ [3] http://pastebin.com/CCFcKq5b [4] http://paste.openstack.org/show/559018/ On Tue, Aug 16, 2016 at 2:41 PM, Erlon Cruz <sombrafam(a)gmail.com&gt; wrote: > Hi Han, > > Thanks for helping. What is the XML before the snapshot is created? Can > you post the XML files you used too? The files are in the links pointed by > the numbers [1], [2] etc. > > The versions are: > qemu-system-x86_64 --version > QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.4), Copyright > (c) 2003-2008 Fabrice Bellard > > libvirtd --version > libvirtd (libvirt) 1.3.1 > > > > Erlon > > On Mon, Aug 15, 2016 at 12:56 AM, Han Han <hhan(a)redhat.com&gt; wrote: > >> Hi，Erlon >> I was not able to reproduce your problem. It is ok to attach disk with >> external snapshot. >> I did it with following steps: >> # qemu-img create /tmp/vdb 100M >> Formatting '/tmp/vdb', fmt=raw size=104857600 >> # virsh start snap >> Domain snap started >> # virsh snapshot-create-as snap s1 --disk-only --diskspec >> vdb,file=/tmp/vdb.s1 --quiesce >> Domain snapshot s1 created >> # virsh dumpxml snap|awk '/<disk/,/<\/disk/' >> <disk type='file' device='disk'> >> <driver name='qemu' type='qcow2' cache='none'/> >> <source file='/var/lib/libvirt/images/snap.s1'/> >> <backingStore type='file' index='1'> >> <format type='qcow2'/> >> <source file='/var/lib/libvirt/images/snap.qcow2'/> >> <backingStore/> >> </backingStore> >> <target dev='vda' bus='virtio'/> >> <alias name='virtio-disk0'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' >> function='0x0'/> >> </disk> >> <disk type='file' device='disk'> >> <driver name='qemu' type='qcow2'/> >> <source file='/tmp/vdb.s1'/> >> <backingStore type='file' index='1'> >> <format type='raw'/> >> <source file='/tmp/vdb'/> >> <backingStore/> >> </backingStore> >> <target dev='vdb' bus='virtio'/> >> <alias name='virtio-disk1'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' >> function='0x0'/> >> </disk> >> >> Save the dumped vdb's xml to /tmp/vdb.xml >> # virsh detach-device snap /tmp/vdb.xml >> Device detached successfully >> # virsh attach-device snap /tmp/vdb.xml >> Device attached successfully >> # virsh dumpxml snap|awk '/<disk/,/<\/disk/' >> <disk type='file' device='disk'> >> <driver name='qemu' type='qcow2' cache='none'/> >> <source file='/var/lib/libvirt/images/snap.s1'/> >> <backingStore type='file' index='1'> >> <format type='qcow2'/> >> <source file='/var/lib/libvirt/images/snap.qcow2'/> >> <backingStore/> >> </backingStore> >> <target dev='vda' bus='virtio'/> >> <alias name='virtio-disk0'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' >> function='0x0'/> >> </disk> >> <disk type='file' device='disk'> >> <driver name='qemu' type='qcow2'/> >> <source file='/tmp/vdb.s1'/> >> <backingStore type='file' index='1'> >> <format type='raw'/> >> <source file='/tmp/vdb'/> >> <backingStore/> >> </backingStore> >> <target dev='vdb' bus='virtio'/> >> <alias name='virtio-disk1'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' >> function='0x0'/> >> </disk> >> >> So I managed to attach disks with snapshot. >> I want to know what is your libvirt and qemu-kvm version? >> And can you provide your files: /tmp/disk.xml[1], >> /tmp/snap-from-disk.xml[2], /tmp/disk-with-snap.xml[4], >> /tmp/disk-with-snap.xml . I need more details. >> >> Han Han >> >> ----- Original Message ----- >> From: "Erlon Cruz" <sombrafam(a)gmail.com&gt; >> To: libvirt-users(a)redhat.com >> Sent: Friday, August 12, 2016 10:23:24 AM >> Subject: [libvirt-users] Attaching disks with external snapshots >> >> Hi folks, >> >> I'm trying to to attach a disk to an instance using libvirt. The problem >> is, this disk has external snapshots. The process tried was: >> >> 1 - Attach a disk in the domain: >> virsh# attach-device instance-00000006 /tmp/disk.xml[1] --live >> >> 2 - Snapshot the disk[2]: >> virsh# snapshot-create instance-00000006 --quiesce --xmlfile >> /tmp/snap-from-disk.xml[2] --disk-only >> >> 3 - Dump the domain XML and create a new disk file from it: >> virsh# dumpxml instance-00000006 >> ... [3] >> >> 4 - Dettach the device and re-add it using the new disk file: >> virsh# detach-device instance-00000006 /tmp/disk-with-snap.xml[4] >> Device detached successfully >> virsh# attach-device instance-00000006 /tmp/disk-with-snap.xml >> error: Failed to attach device from /tmp/disk-with-snap.xml >> error: internal error: unable to execute QEMU command 'device_add': >> Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1' >> >> Question, is this operation supported? If yes, how is the correct >> procedure? >> >> Erlon >> >> >> [1] http://paste.openstack.org/show/556055/ >> [2] http://paste.openstack.org/show/556056/ >> [3] http://paste.openstack.org/show/556063/ >> [4] http://paste.openstack.org/show/556064/ >> >> _______________________________________________ >> libvirt-users mailing list >> libvirt-users(a)redhat.com >> https://www.redhat.com/mailman/listinfo/libvirt-users >> > >

If that was the case I wouldn't be able to attach a volume without snapshots. But just in case I did the following test, before attaching the file, I copied all the file to a local folder and changed the path in XML with the new paths. The problem repeats: *2016-08-22 17:14:19.795+0000: 3799: info : qemuMonitorIOProcess:423 : QEMU_MONITOR_IO_PROCESS: mon=0x7f68f800c4c0 buf={"return": "Could not open backing file: Could not open '/home/ubuntu/volumes/volume-d81f7e7e-4215-4433-aad2-50adcd31b29d.7a9ea169-a3ab-48bd-bd9d-e7585860abe4': Permission denied\r\n", "id": "libvirt-30"}* But, if I use the top backing file with same permissions (/home/ubuntu/volumes/volume-d81f7e7e-4215-4433-aad2-50adcd31b29d), the operation suceeds. Can this be a libvirt/QEMU bug? Han, can you tell me more details about the commands you used and the versions of or libvirt/qemu? I couldn't reproduce from the steps you outlined. On Fri, Aug 19, 2016 at 12:02 AM, Han Han <hhan(a)redhat.com&gt; wrote: > Hi Erlon, > Since you say you got Permission denied error while the ACL mode is 777， > I guess it is due to security system like selinux or apparmor. > As I know, selinux defaultly will disable qemu to use volume based on > NFS. We need to use 'setsebool virt_use_nfs 1' to enable it. > Well, your OS is ubuntu, so I think apparmor causes the permission > problem. Try to find a way to enable virt_use_nfs or disable apparmor. Then > check if the problem can be reproduced. > > Han > > ----- Original Message ----- > From: "Erlon Cruz" <sombrafam(a)gmail.com&gt; > To: "Han Han" <hhan(a)redhat.com&gt; > Cc: "Yang Yang" <yanyang(a)redhat.com&gt;, "Yuan Dan" <dyuan(a)redhat.com&gt;, > "Xuesong Zhang" <xuzhang(a)redhat.com&gt;, libvirt-users(a)redhat.com > Sent: Thursday, 18 August, 2016 12:44:36 PM > Subject: Re: [libvirt-users] Attaching disks with external snapshots > > Hi Han, > > Any update on this? > > Erlon > > On Wed, Aug 17, 2016 at 9:58 AM, Erlon Cruz <sombrafam(a)gmail.com&gt; wrote: > > > Han, > > > > Did the same test again with libvirt debug enabled. This time I added > the > > disk[1] without snaps (OK), and then added the disk[2] with snapshots. > The > > libvirt log[3] (line 320) says: > > > > *Could not open > > '/opt/stack/data/nova/mnt/a3b4c6ddd9bf82edd4f726872be58d05/ > volume-a75b42e8-8175-4516-b724-43ee6f3d2520': > > Permission denied\r\n", "id": "libvirt-33"}]* > > > > But this file has 777 permissions. I can access it with any user I > > tried[4]. Tough its mounted over NFS, the permissions seems to be right. > > > > Erlon > > > > [1] http://paste.openstack.org/show/556055/ > > [2] http://paste.openstack.org/show/556056/ > > [3] http://pastebin.com/CCFcKq5b > > [4] http://paste.openstack.org/show/559018/ > > > > > > On Tue, Aug 16, 2016 at 2:41 PM, Erlon Cruz <sombrafam(a)gmail.com&gt; > wrote: > > > >> Hi Han, > >> > >> Thanks for helping. What is the XML before the snapshot is created? Can > >> you post the XML files you used too? The files are in the links > pointed by > >> the numbers [1], [2] etc. > >> > >> The versions are: > >> qemu-system-x86_64 --version > >> QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.4), Copyright > >> (c) 2003-2008 Fabrice Bellard > >> > >> libvirtd --version > >> libvirtd (libvirt) 1.3.1 > >> > >> > >> > >> Erlon > >> > >> On Mon, Aug 15, 2016 at 12:56 AM, Han Han <hhan(a)redhat.com&gt; wrote: > >> > >>> Hi，Erlon > >>> I was not able to reproduce your problem. It is ok to attach disk with > >>> external snapshot. > >>> I did it with following steps: > >>> # qemu-img create /tmp/vdb 100M > >>> Formatting '/tmp/vdb', fmt=raw size=104857600 > >>> # virsh start snap > >>> Domain snap started > >>> # virsh snapshot-create-as snap s1 --disk-only --diskspec > >>> vdb,file=/tmp/vdb.s1 --quiesce > >>> Domain snapshot s1 created > >>> # virsh dumpxml snap|awk '/<disk/,/<\/disk/' > >>> <disk type='file' device='disk'> > >>> <driver name='qemu' type='qcow2' cache='none'/> > >>> <source file='/var/lib/libvirt/images/snap.s1'/> > >>> <backingStore type='file' index='1'> > >>> <format type='qcow2'/> > >>> <source file='/var/lib/libvirt/images/snap.qcow2'/> > >>> <backingStore/> > >>> </backingStore> > >>> <target dev='vda' bus='virtio'/> > >>> <alias name='virtio-disk0'/> > >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' > >>> function='0x0'/> > >>> </disk> > >>> <disk type='file' device='disk'> > >>> <driver name='qemu' type='qcow2'/> > >>> <source file='/tmp/vdb.s1'/> > >>> <backingStore type='file' index='1'> > >>> <format type='raw'/> > >>> <source file='/tmp/vdb'/> > >>> <backingStore/> > >>> </backingStore> > >>> <target dev='vdb' bus='virtio'/> > >>> <alias name='virtio-disk1'/> > >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' > >>> function='0x0'/> > >>> </disk> > >>> > >>> Save the dumped vdb's xml to /tmp/vdb.xml > >>> # virsh detach-device snap /tmp/vdb.xml > >>> Device detached successfully > >>> # virsh attach-device snap /tmp/vdb.xml > >>> Device attached successfully > >>> # virsh dumpxml snap|awk '/<disk/,/<\/disk/' > >>> <disk type='file' device='disk'> > >>> <driver name='qemu' type='qcow2' cache='none'/> > >>> <source file='/var/lib/libvirt/images/snap.s1'/> > >>> <backingStore type='file' index='1'> > >>> <format type='qcow2'/> > >>> <source file='/var/lib/libvirt/images/snap.qcow2'/> > >>> <backingStore/> > >>> </backingStore> > >>> <target dev='vda' bus='virtio'/> > >>> <alias name='virtio-disk0'/> > >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' > >>> function='0x0'/> > >>> </disk> > >>> <disk type='file' device='disk'> > >>> <driver name='qemu' type='qcow2'/> > >>> <source file='/tmp/vdb.s1'/> > >>> <backingStore type='file' index='1'> > >>> <format type='raw'/> > >>> <source file='/tmp/vdb'/> > >>> <backingStore/> > >>> </backingStore> > >>> <target dev='vdb' bus='virtio'/> > >>> <alias name='virtio-disk1'/> > >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' > >>> function='0x0'/> > >>> </disk> > >>> > >>> So I managed to attach disks with snapshot. > >>> I want to know what is your libvirt and qemu-kvm version? > >>> And can you provide your files: /tmp/disk.xml[1], > >>> /tmp/snap-from-disk.xml[2], /tmp/disk-with-snap.xml[4], > >>> /tmp/disk-with-snap.xml . I need more details. > >>> > >>> Han Han > >>> > >>> ----- Original Message ----- > >>> From: "Erlon Cruz" <sombrafam(a)gmail.com&gt; > >>> To: libvirt-users(a)redhat.com > >>> Sent: Friday, August 12, 2016 10:23:24 AM > >>> Subject: [libvirt-users] Attaching disks with external snapshots > >>> > >>> Hi folks, > >>> > >>> I'm trying to to attach a disk to an instance using libvirt. The > problem > >>> is, this disk has external snapshots. The process tried was: > >>> > >>> 1 - Attach a disk in the domain: > >>> virsh# attach-device instance-00000006 /tmp/disk.xml[1] --live > >>> > >>> 2 - Snapshot the disk[2]: > >>> virsh# snapshot-create instance-00000006 --quiesce --xmlfile > >>> /tmp/snap-from-disk.xml[2] --disk-only > >>> > >>> 3 - Dump the domain XML and create a new disk file from it: > >>> virsh# dumpxml instance-00000006 > >>> ... [3] > >>> > >>> 4 - Dettach the device and re-add it using the new disk file: > >>> virsh# detach-device instance-00000006 /tmp/disk-with-snap.xml[4] > >>> Device detached successfully > >>> virsh# attach-device instance-00000006 /tmp/disk-with-snap.xml > >>> error: Failed to attach device from /tmp/disk-with-snap.xml > >>> error: internal error: unable to execute QEMU command 'device_add': > >>> Property 'virtio-blk-device.drive' can't find value > 'drive-virtio-disk1' > >>> > >>> Question, is this operation supported? If yes, how is the correct > >>> procedure? > >>> > >>> Erlon > >>> > >>> > >>> [1] http://paste.openstack.org/show/556055/ > >>> [2] http://paste.openstack.org/show/556056/ > >>> [3] http://paste.openstack.org/show/556063/ > >>> [4] http://paste.openstack.org/show/556064/ > >>> > >>> _______________________________________________ > >>> libvirt-users mailing list > >>> libvirt-users(a)redhat.com > >>> https://www.redhat.com/mailman/listinfo/libvirt-users > >>> > >> > >> > > >