On Tue, 2015-06-30 at 17:51 +0100, Daniel P. Berrange wrote:

The libc.so library requires certainly functionality to be provided by the ld-linux.so library. The ld-linux.so is loaded by the kernel based on the PT_INTERP ELF section, and as such LD_LIBRARY_PATH has no effect.

So, although libvirt-sandbox-init-{qemu,lxc} set LD_LIBRARY_PATH to force use of the libs from the host OS image, the common init program was stil using the ld-linux.so from the guest OS image. Sometimes this works, sometimes it breaks.

When trying to run Fedora 19 as the root filesystem from a Fedora 22 host, it would break thus:

/etc/libvirt-sandbox/scratch/.libs/libvirt-sandbox-init-common: relocation error: /etc/libvirt-sandbox/scratch/.libs/libc.so.6: symbol _dl_find_dso_for_object, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

To deal with this we must ensure that we always invoke the libvirt-sandbox-init-common program using the ld-linux that was provided by the host OS image.

The sandbox builder is tweaked so that it always copies the host ld-linux.so into the libs scratch directory, and gives it a predictable name 'ld.so', since every architecture seems to have a different name.

The libvirt-sandbox-init-{lxc,qemu} files are changed so that instead of exec'ing libvirt-sandbox-init-common directly, they will load it via the ld.so file. --- libvirt-sandbox/libvirt-sandbox-builder.c | 28 +++++++++++++++++++++++----- libvirt-sandbox/libvirt-sandbox-init-lxc.c | 1 + libvirt-sandbox/libvirt-sandbox-init-qemu.c | 1 + 3 files changed, 25 insertions(+), 5 deletions(-)

diff --git a/libvirt-sandbox/libvirt-sandbox-builder.c b/libvirt-sandbox/libvirt-sandbox-builder.c index 2726868..e615606 100644 --- a/libvirt-sandbox/libvirt-sandbox-builder.c +++ b/libvirt-sandbox/libvirt-sandbox-builder.c @@ -252,10 +252,11 @@ GVirConnection *gvir_sandbox_builder_get_connection(GVirSandboxBuilder *builder)

static gboolean gvir_sandbox_builder_copy_file(const char *path, const char *libsdir, + const char *newname, GError **error) { gchar *name = g_path_get_basename(path); - gchar *target = g_build_filename(libsdir, name, NULL); + gchar *target = g_build_filename(libsdir, newname ? newname : name, NULL); GFile *srcFile = g_file_new_for_path(path); GFile *tgtFile = g_file_new_for_path(target); gboolean result = FALSE; @@ -285,7 +286,7 @@ static gboolean gvir_sandbox_builder_copy_program(const char *program, const gchar *argv[] = {LDD_PATH, program, NULL}; gboolean result = FALSE;

- if (!gvir_sandbox_builder_copy_file(program, dest, error)) + if (!gvir_sandbox_builder_copy_file(program, dest, NULL, error)) goto cleanup;

@@ -301,14 +302,31 @@ static gboolean gvir_sandbox_builder_copy_program(const char *program, *tmp = '\0';

/* Search the line for the library path */ - start = strstr(line, " => "); + start = strstr(line, "/"); end = strstr(line, " (");

if (start && end) { - start = start + 4; + const gchar *newname = NULL; *end = '\0';

- if (!gvir_sandbox_builder_copy_file(start, dest, error)) + /* There are countless different naming schemes for + * the ld-linux.so library across architectures. Pretty + * much the only thing in common is they start with + * the two letters 'ld'. The LDD program prints it + * out differently too - it doesn't include " => " + * as this library is special - its actually a static + * linked executable not a library. + * + * To make life easier for libvirt-sandbox-init-{qemu,lxc} + * we just call the file 'ld.so' when we copy it into our + * scratch dir, no matter what it was called on the host. + */ + if (!strstr(line, " => ") && + strstr(start, "/ld")) { + newname = "ld.so"; + } + + if (!gvir_sandbox_builder_copy_file(start, dest, newname, error)) goto cleanup; }

diff --git a/libvirt-sandbox/libvirt-sandbox-init-lxc.c b/libvirt-sandbox/libvirt-sandbox-init-lxc.c index e2fe7f0..d1e4a79 100644 --- a/libvirt-sandbox/libvirt-sandbox-init-lxc.c +++ b/libvirt-sandbox/libvirt-sandbox-init-lxc.c @@ -77,6 +77,7 @@ main(int argc, char **argv) args[narg++] = "1000"; }

+ args[narg++] = SANDBOXCONFIGDIR "/.libs/ld.so"; args[narg++] = SANDBOXCONFIGDIR "/.libs/libvirt-sandbox-init-common"; if (debug) args[narg++] = "-d"; diff --git a/libvirt-sandbox/libvirt-sandbox-init-qemu.c b/libvirt-sandbox/libvirt-sandbox-init-qemu.c index 09580da..cd6055a 100644 --- a/libvirt-sandbox/libvirt-sandbox-init-qemu.c +++ b/libvirt-sandbox/libvirt-sandbox-init-qemu.c @@ -422,6 +422,7 @@ main(int argc ATTR_UNUSED, char **argv ATTR_UNUSED) args[narg++] = "1000"; }

+ args[narg++] = SANDBOXCONFIGDIR "/.libs/ld.so"; args[narg++] = SANDBOXCONFIGDIR "/.libs/libvirt-sandbox-init-common"; if (debug) args[narg++] = "-d";

ACK. Tested it with a sles11 sp3 guest on openSUSE 13.2: works like a charm. -- Cedric