Re: [libvirt] Stored secrets seem to get corrupted
On 06/25/2012 04:54 PM, Daniel P. Berrange wrote:
On Mon, Jun 25, 2012 at 04:37:48PM +0200, Wido den Hollander wrote:
> Hi,
>
> On one of my systems I'm having troubles with my RBD storage backend.
>
> At first I thought it was a problem with my code, but after trying
> the same code on a second machine I'm a bit confused.
>
> The problem is that the storage backend tries to retrieve the value
> of a secret and base64 decode it, that fails.
>
> My debug log shows:
> Notice this behavior:
>
> root@stack01:~# virsh secret-set-value
> 322bccea-f2ed-4eae-a7e5-d0793ffb162d
> AQAE+uJPCFpELBAAkTniQvHabBGj0Quwnu2imA==
> Secret value set
>
> root@stack01:~# md5sum
> /etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
> b4b147bc522828731f1a016bfa72c073
> /etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
> root@stack01:~# virsh secret-set-value
> 322bccea-f2ed-4eae-a7e5-d0793ffb162d
> AQAE+uJPCFpELBAAkTniQvHabBGj0Quwnu2imA==
> Secret value set
>
> root@stack01:~# md5sum
> /etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
> 927e2458c32cc3f6754d91694e41333f
> /etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64
> root@stack01:~#
>
> As you can see, the md5sum of the file changes when I set the value
> of the secret to the same.
That is really bizarre. Can you look at what is actually stored
in the .base64 file each time ? And what 'secret-get-value'
replies with ?
The content of the .base64 is pure garbage, my terminal can't make
anything of it.
What I do notice is that the .base64 file is only 2 bytes big, while it
should be 40 bytes.
"secret-get-value" returns the correct data, but I think that is due to
it being in memory. That also tells me that the writing to disk fails,
in memory it is still fine.
When I restart libvirt I see:
secretLoadValue:406 : internal error invalid base64 in
'/etc/libvirt/secrets/322bccea-f2ed-4eae-a7e5-d0793ffb162d.base64'
I checked my disk-space and inodes again, but those are all fine. I can
write other files on the same FS without any problem.
I also made sure that AppArmor (Ubuntu) was turned off.
> I verified that stack01 isn't out of disk space or out of
inodes,
> those are in the acceptable values range.
>
> Any suggestions?
I think you'll probably need to add some more VIR_DEBUG lines
to secret_driver.c to see where in the process it is going
wrong. Or perhaps strace libvirtd to see what it thinks it
is writing out & whether any errors appear.
I'll try that.
Wido