5 a.m.
I'm trying to enable CI coverage for macOS 12, but I'm running into a
couple of issues that I'm not sure how to handle.
Note that the test suite currently passes on macOS 11[1], so these
failures have to be a consequence to changes made to macOS that we
haven't yet learned how to cope with.
The first one is in vircryptotest:
Encrypt aes265cbc ... Expected ciphertext doesn't match
I've added some debug statements and it looks like the generated data
is different every time, which seems like a pretty good indication
that virrandommock is not being picked up correctly. This is not the
only test program that uses that specific mock though, so I'm not
sure what makes it fail when all the others are succeeding.
The other issue is in qemuxml2argvtest:
error : virCommandWait:2752 : internal error: Child process
(/usr/libexec/qemu/vhost-user/test-vhost-user-gpu --print-capabilities)
unexpected fatal signal 6: dyld[8896]: symbol not found in flat
namespace '_virQEMUCapsGet'
error : qemuVhostUserFillDomainGPU:394 : operation failed: Unable to
find a satisfying vhost-user-gpu
So the various virFileWrapperAddPrefix() calls that cause the
contents of tests/qemuvhostuserdata/ to override the host's own
vhostuser configuration are still effective, but for some reason the
trivial test-vhost-user-gpu shell script can't be run successfully
because an internal libvirt symbol can't be found somehow? Confusing.
Roman, does any of this ring a bell? Any chance you could
investigate? macOS 12 has been out for a while now so I'd be very
keen to have it added to CI.
Thanks in advance!
[1] https://gitlab.com/libvirt/libvirt/-/jobs/2421455154
--
Andrea Bolognani / Red Hat / Virtualization
5:10 a.m.
On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote:
I'm trying to enable CI coverage for macOS 12, but I'm
running into a
couple of issues that I'm not sure how to handle.
Note that the test suite currently passes on macOS 11[1], so these
failures have to be a consequence to changes made to macOS that we
haven't yet learned how to cope with.
Our macOS 11 config askes for 'big-sur-base' and based on logs our
that is getting clang 12.5.1.
Cirrus CI jobs indicate you can request a 'big-sur-xcode-13' which
gives newer clang. So perhaps try testing that image in our config,
to narrow down whether the problem is macOS 12 vs clang >= 12
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
7:35 a.m.
On Fri, May 06, 2022 at 11:10:44AM +0100, Daniel P. Berrangé wrote:
On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote:
> I'm trying to enable CI coverage for macOS 12, but I'm running into a
> couple of issues that I'm not sure how to handle.
>
> Note that the test suite currently passes on macOS 11[1], so these
> failures have to be a consequence to changes made to macOS that we
> haven't yet learned how to cope with.
Our macOS 11 config askes for 'big-sur-base' and based on logs our
that is getting clang 12.5.1.
Cirrus CI jobs indicate you can request a 'big-sur-xcode-13' which
gives newer clang. So perhaps try testing that image in our config,
to narrow down whether the problem is macOS 12 vs clang >= 12
Interesting idea! Unfortunately the test suite still passes on macOS
11 even when using Clang 13, so it looks like something really
changed at the OS level.
https://gitlab.com/abologna/libvirt/-/pipelines/532966833
--
Andrea Bolognani / Red Hat / Virtualization
4:34 a.m.
On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote:
I'm trying to enable CI coverage for macOS 12, but I'm
running into a
couple of issues that I'm not sure how to handle.
Note that the test suite currently passes on macOS 11[1], so these
failures have to be a consequence to changes made to macOS that we
haven't yet learned how to cope with.
The first one is in vircryptotest:
Encrypt aes265cbc ... Expected ciphertext doesn't match
I've added some debug statements and it looks like the generated data
is different every time, which seems like a pretty good indication
that virrandommock is not being picked up correctly. This is not the
only test program that uses that specific mock though, so I'm not
sure what makes it fail when all the others are succeeding.
The other issue is in qemuxml2argvtest:
error : virCommandWait:2752 : internal error: Child process
(/usr/libexec/qemu/vhost-user/test-vhost-user-gpu --print-capabilities)
unexpected fatal signal 6: dyld[8896]: symbol not found in flat
namespace '_virQEMUCapsGet'
error : qemuVhostUserFillDomainGPU:394 : operation failed: Unable to
find a satisfying vhost-user-gpu
So the various virFileWrapperAddPrefix() calls that cause the
contents of tests/qemuvhostuserdata/ to override the host's own
vhostuser configuration are still effective, but for some reason the
trivial test-vhost-user-gpu shell script can't be run successfully
because an internal libvirt symbol can't be found somehow? Confusing.
Roman, does any of this ring a bell? Any chance you could
investigate? macOS 12 has been out for a while now so I'd be very
keen to have it added to CI.
Roman, any chance you can find some time to investigate the test
failures documented above? I'm afraid I've reached the limit of my
ability to debug macOS-specific behavior :(
--
Andrea Bolognani / Red Hat / Virtualization
1:19 p.m.
Hi Andrea,
I saw your call to Sergio and Marc-André on IRC, and I thought I would
spend a few minutes inviestigating.
On 10 Jun 2022, at 11:34, Andrea Bolognani
<abologna(a)redhat.com> wrote:
On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote:
> I'm trying to enable CI coverage for macOS 12, but I'm running into a
> couple of issues that I'm not sure how to handle.
>
> Note that the test suite currently passes on macOS 11[1], so these
> failures have to be a consequence to changes made to macOS that we
> haven't yet learned how to cope with.
>
> The first one is in vircryptotest:
>
> Encrypt aes265cbc ... Expected ciphertext doesn't match
>
> I've added some debug statements and it looks like the generated data
> is different every time, which seems like a pretty good indication
> that virrandommock is not being picked up correctly. This is not the
> only test program that uses that specific mock though, so I'm not
> sure what makes it fail when all the others are succeeding.
I believe that the following patch fixes this one:
From: Christophe de Dinechin <christophe(a)dinechin.org>
Date: Mon, 8 Aug 2022 20:14:08 +0200
Subject: [PATCH] tests: Pass the flat_namespace option to the linker
This fixes vircryptotest on macOS 12 (Monterey).
The test relies on library injection (using DYLD_INSERT_LIBRARIES)
to replace the normal random functions with functions giving predictable
results, defined in virrandommock.c. However, using DYLD_INSERT_LIBRARIES
only works when building with flat namespaces.
Adding the -Wl,-flat_namespace option to the linker fixes the problem.
The option was already defined in the top-level meson.build, but had been
forgotten in the test linker arguments.
Signed-off-by: Christophe de Dinechin <dinechin(a)redhat.com>
Signed-off-by: Christophe de Dinechin <christophe(a)dinechin.org>
---
tests/meson.build | 1 +
1 file changed, 1 insertion(+)
diff --git a/tests/meson.build b/tests/meson.build
index bc9d8ccc4c..d6b1bb2bf0 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -28,6 +28,7 @@ tests_dep = declare_dependency(
],
link_args: (
libvirt_export_dynamic
+ + libvirt_flat_namespace
+ coverage_flags
),
)
--
2.37.1
Could you please check?
>
> The other issue is in qemuxml2argvtest:
>
> error : virCommandWait:2752 : internal error: Child process
> (/usr/libexec/qemu/vhost-user/test-vhost-user-gpu --print-capabilities)
> unexpected fatal signal 6: dyld[8896]: symbol not found in flat
> namespace '_virQEMUCapsGet'
> error : qemuVhostUserFillDomainGPU:394 : operation failed: Unable to
> find a satisfying vhost-user-gpu
>
> So the various virFileWrapperAddPrefix() calls that cause the
> contents of tests/qemuvhostuserdata/ to override the host's own
> vhostuser configuration are still effective, but for some reason the
> trivial test-vhost-user-gpu shell script can't be run successfully
> because an internal libvirt symbol can't be found somehow? Confusing.
>
> Roman, does any of this ring a bell? Any chance you could
> investigate? macOS 12 has been out for a while now so I'd be very
> keen to have it added to CI.
Roman, any chance you can find some time to investigate the test
failures documented above? I'm afraid I've reached the limit of my
ability to debug macOS-specific behavior :(
--
Andrea Bolognani / Red Hat / Virtualization
4:28 a.m.
On Mon, Aug 08, 2022 at 08:19:28PM +0200, Christophe de Dinechin wrote:
Hi Andrea,
I saw your call to Sergio and Marc-André on IRC, and I thought I would
spend a few minutes inviestigating.
Thanks, I appreciate that!
>> I'm trying to enable CI coverage for macOS 12, but
I'm running into a
>> couple of issues that I'm not sure how to handle.
>>
>> Note that the test suite currently passes on macOS 11[1], so these
>> failures have to be a consequence to changes made to macOS that we
>> haven't yet learned how to cope with.
>>
>> The first one is in vircryptotest:
>>
>> Encrypt aes265cbc ... Expected ciphertext doesn't match
>>
>> I've added some debug statements and it looks like the generated data
>> is different every time, which seems like a pretty good indication
>> that virrandommock is not being picked up correctly. This is not the
>> only test program that uses that specific mock though, so I'm not
>> sure what makes it fail when all the others are succeeding.
I believe that the following patch fixes this one:
From: Christophe de Dinechin <christophe(a)dinechin.org>
Date: Mon, 8 Aug 2022 20:14:08 +0200
Subject: [PATCH] tests: Pass the flat_namespace option to the linker
This fixes vircryptotest on macOS 12 (Monterey).
The test relies on library injection (using DYLD_INSERT_LIBRARIES)
to replace the normal random functions with functions giving predictable
results, defined in virrandommock.c. However, using DYLD_INSERT_LIBRARIES
only works when building with flat namespaces.
Adding the -Wl,-flat_namespace option to the linker fixes the problem.
The option was already defined in the top-level meson.build, but had been
forgotten in the test linker arguments.
Signed-off-by: Christophe de Dinechin <dinechin(a)redhat.com>
Signed-off-by: Christophe de Dinechin <christophe(a)dinechin.org>
---
tests/meson.build | 1 +
1 file changed, 1 insertion(+)
diff --git a/tests/meson.build b/tests/meson.build
index bc9d8ccc4c..d6b1bb2bf0 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -28,6 +28,7 @@ tests_dep = declare_dependency(
],
link_args: (
libvirt_export_dynamic
+ + libvirt_flat_namespace
+ coverage_flags
),
)
--
2.37.1
Could you please check?
Yeah, this seems to help and the change makes sense to me.
I wonder why we didn't run into this much earlier though? As I
mentioned, the test runs successfully as-is on macOS 11. Plus, many
other tests rely on library injection and yet work okay even without
this change.
Anyway, I'm happy to add my
Reviewed-by: Andrea Bolognani <abologna(a)redhat.com>
to this patch and push it. The authorship information looks a bit
funky though, with the two S-o-bs... Should I drop the @redhat.com
one and only keep the one with your personal email address, matching
the commit authorship information?
--
Andrea Bolognani / Red Hat / Virtualization
4:34 a.m.
On 9 Aug 2022, at 11:28, Andrea Bolognani <abologna(a)redhat.com>
wrote:
On Mon, Aug 08, 2022 at 08:19:28PM +0200, Christophe de Dinechin wrote:
> Hi Andrea,
>
> I saw your call to Sergio and Marc-André on IRC, and I thought I would
> spend a few minutes inviestigating.
Thanks, I appreciate that!
>>> I'm trying to enable CI coverage for macOS 12, but I'm running into
a
>>> couple of issues that I'm not sure how to handle.
>>>
>>> Note that the test suite currently passes on macOS 11[1], so these
>>> failures have to be a consequence to changes made to macOS that we
>>> haven't yet learned how to cope with.
>>>
>>> The first one is in vircryptotest:
>>>
>>> Encrypt aes265cbc ... Expected ciphertext doesn't match
>>>
>>> I've added some debug statements and it looks like the generated data
>>> is different every time, which seems like a pretty good indication
>>> that virrandommock is not being picked up correctly. This is not the
>>> only test program that uses that specific mock though, so I'm not
>>> sure what makes it fail when all the others are succeeding.
>
> I believe that the following patch fixes this one:
>
> From: Christophe de Dinechin <christophe(a)dinechin.org>
> Date: Mon, 8 Aug 2022 20:14:08 +0200
> Subject: [PATCH] tests: Pass the flat_namespace option to the linker
>
> This fixes vircryptotest on macOS 12 (Monterey).
>
> The test relies on library injection (using DYLD_INSERT_LIBRARIES)
> to replace the normal random functions with functions giving predictable
> results, defined in virrandommock.c. However, using DYLD_INSERT_LIBRARIES
> only works when building with flat namespaces.
>
> Adding the -Wl,-flat_namespace option to the linker fixes the problem.
> The option was already defined in the top-level meson.build, but had been
> forgotten in the test linker arguments.
>
> Signed-off-by: Christophe de Dinechin <dinechin(a)redhat.com>
> Signed-off-by: Christophe de Dinechin <christophe(a)dinechin.org>
> ---
> tests/meson.build | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/tests/meson.build b/tests/meson.build
> index bc9d8ccc4c..d6b1bb2bf0 100644
> --- a/tests/meson.build
> +++ b/tests/meson.build
> @@ -28,6 +28,7 @@ tests_dep = declare_dependency(
> ],
> link_args: (
> libvirt_export_dynamic
> + + libvirt_flat_namespace
> + coverage_flags
> ),
> )
> --
> 2.37.1
>
>
> Could you please check?
Yeah, this seems to help and the change makes sense to me.
I wonder why we didn't run into this much earlier though? As I
mentioned, the test runs successfully as-is on macOS 11. Plus, many
other tests rely on library injection and yet work okay even without
this change.
Anyway, I'm happy to add my
Reviewed-by: Andrea Bolognani <abologna(a)redhat.com>
to this patch and push it. The authorship information looks a bit
funky though, with the two S-o-bs...
I did not know which one you’d prefer (in case there is a policy).
If I get to choose, assign that to Red Hat (and change the author accordingly).
(and I’ll change my libvirt gitconfig accordingly in the future)
Thanks,
Christophe
Should I drop the @redhat.com
one and only keep the one with your personal email address, matching
the commit authorship information?
--
Andrea Bolognani / Red Hat / Virtualization
4:50 a.m.
On Tue, Aug 09, 2022 at 11:34:20AM +0200, Christophe de Dinechin wrote:
On 9 Aug 2022, at 11:28, Andrea Bolognani <abologna(a)redhat.com>
wrote:
> Yeah, this seems to help and the change makes sense to me.
>
> I wonder why we didn't run into this much earlier though? As I
> mentioned, the test runs successfully as-is on macOS 11. Plus, many
> other tests rely on library injection and yet work okay even without
> this change.
>
> Anyway, I'm happy to add my
>
> Reviewed-by: Andrea Bolognani <abologna(a)redhat.com>
>
> to this patch and push it. The authorship information looks a bit
> funky though, with the two S-o-bs...
I did not know which one you’d prefer (in case there is a policy).
If I get to choose, assign that to Red Hat (and change the author accordingly).
(and I’ll change my libvirt gitconfig accordingly in the future)
Done. I'll push once CI has passed.
It would be great if you could use git-publish for future code
submissions: that way patches can be applied locally more
conveniently by the reviewer. I was able to make it work regardless,
it just took a bit more effort :)
--
Andrea Bolognani / Red Hat / Virtualization
4:59 a.m.
On 9 Aug 2022, at 11:50, Andrea Bolognani <abologna(a)redhat.com>
wrote:
On Tue, Aug 09, 2022 at 11:34:20AM +0200, Christophe de Dinechin wrote:
> On 9 Aug 2022, at 11:28, Andrea Bolognani <abologna(a)redhat.com> wrote:
>> Yeah, this seems to help and the change makes sense to me.
>>
>> I wonder why we didn't run into this much earlier though? As I
>> mentioned, the test runs successfully as-is on macOS 11. Plus, many
>> other tests rely on library injection and yet work okay even without
>> this change.
I must admit that this puzzled me a bit too. I spent a bit of time checking
for dyld warnings or anything else.
One explanation could be if in other cases, the symbols are marked as weak?
I did not check that. And I don’t have a macOS 11 machine to compare anymore.
>>
>> Anyway, I'm happy to add my
>>
>> Reviewed-by: Andrea Bolognani <abologna(a)redhat.com>
>>
>> to this patch and push it. The authorship information looks a bit
>> funky though, with the two S-o-bs...
>
> I did not know which one you’d prefer (in case there is a policy).
> If I get to choose, assign that to Red Hat (and change the author accordingly).
>
> (and I’ll change my libvirt gitconfig accordingly in the future)
Done. I'll push once CI has passed.
It would be great if you could use git-publish for future code
submissions: that way patches can be applied locally more
conveniently by the reviewer. I was able to make it work regardless,
it just took a bit more effort :)
Ack.
https://gitlab.com/c3d/libvirt/-/pipelines/608168172
--
Andrea Bolognani / Red Hat / Virtualization
1:54 p.m.
On 10 Jun 2022, at 11:34, Andrea Bolognani
<abologna(a)redhat.com> wrote:
On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote:
> I'm trying to enable CI coverage for macOS 12, but I'm running into a
> couple of issues that I'm not sure how to handle.
>
> Note that the test suite currently passes on macOS 11[1], so these
> failures have to be a consequence to changes made to macOS that we
> haven't yet learned how to cope with.
>
> The first one is in vircryptotest:
>
> Encrypt aes265cbc ... Expected ciphertext doesn't match
>
> I've added some debug statements and it looks like the generated data
> is different every time, which seems like a pretty good indication
> that virrandommock is not being picked up correctly. This is not the
> only test program that uses that specific mock though, so I'm not
> sure what makes it fail when all the others are succeeding.
>
> The other issue is in qemuxml2argvtest:
>
> error : virCommandWait:2752 : internal error: Child process
> (/usr/libexec/qemu/vhost-user/test-vhost-user-gpu --print-capabilities)
> unexpected fatal signal 6: dyld[8896]: symbol not found in flat
> namespace '_virQEMUCapsGet'
> error : qemuVhostUserFillDomainGPU:394 : operation failed: Unable to
> find a satisfying vhost-user-gpu
I don’t see this one.
What I see instead is failures on qemucapabilitiestest.
The tests seems to depend on a particular install path for the
qemu-system- binaries:
binary = g_strdup_printf("/usr/bin/qemu-system-%s",
data->archName);
On macOS, there is something called “system integrity protection”
which makes it a bit more difficult to add stuff to /usr/bin.
However, I don’t really think that’s the problem. I ran some tests with:
VIR_TEST_DEBUG=1
VIR_TEST_RANGE=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69
/Users/ddd/Work/libvirt/build/tests/qemucapabilitiestest
The error I get seem to point to ‘hvf’ been unexected in the output:
15) 6.2.0 (x86_64) ...
In '/Users/ddd/Work/libvirt/tests/qemucapabilitiesdata/caps_6.2.0.x86_64.xml':
Offset 7557
Expect [c]
Actual [hvf'/>
<flag name=‘c]
I do not really understand what the test is supposed to do. It seems
to be comparing with reference files, but where does it get the
qemu capabilities to compare with from?
How should I proceed?
>
> So the various virFileWrapperAddPrefix() calls that cause the
> contents of tests/qemuvhostuserdata/ to override the host's own
> vhostuser configuration are still effective, but for some reason the
> trivial test-vhost-user-gpu shell script can't be run successfully
> because an internal libvirt symbol can't be found somehow? Confusing.
>
> Roman, does any of this ring a bell? Any chance you could
> investigate? macOS 12 has been out for a while now so I'd be very
> keen to have it added to CI.
Roman, any chance you can find some time to investigate the test
failures documented above? I'm afraid I've reached the limit of my
ability to debug macOS-specific behavior :(
--
Andrea Bolognani / Red Hat / Virtualization
9:41 a.m.
On Mon, Aug 08, 2022 at 08:54:14PM +0200, Christophe de Dinechin wrote:
On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote:
> The other issue is in qemuxml2argvtest:
>
> error : virCommandWait:2752 : internal error: Child process
> (/usr/libexec/qemu/vhost-user/test-vhost-user-gpu --print-capabilities)
> unexpected fatal signal 6: dyld[8896]: symbol not found in flat
> namespace '_virQEMUCapsGet'
> error : qemuVhostUserFillDomainGPU:394 : operation failed: Unable to
> find a satisfying vhost-user-gpu
I don’t see this one.
Dang :(
What I see instead is failures on qemucapabilitiestest.
The tests seems to depend on a particular install path for the
qemu-system- binaries:
binary = g_strdup_printf("/usr/bin/qemu-system-%s",
data->archName);
On macOS, there is something called “system integrity protection”
which makes it a bit more difficult to add stuff to /usr/bin.
However, I don’t really think that’s the problem.
We definitely shouldn't be looking at the contents of the build
machine's actual /usr/bin directory in our test suite! That'd be a
bug in our mocking machinery.
I ran some tests with:
VIR_TEST_DEBUG=1
VIR_TEST_RANGE=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69
/Users/ddd/Work/libvirt/build/tests/qemucapabilitiestest
The error I get seem to point to ‘hvf’ been unexected in the output:
15) 6.2.0 (x86_64) ...
In '/Users/ddd/Work/libvirt/tests/qemucapabilitiesdata/caps_6.2.0.x86_64.xml':
Offset 7557
Expect [c]
Actual [hvf'/>
<flag name=‘c]
I do not really understand what the test is supposed to do. It seems
to be comparing with reference files, but where does it get the
qemu capabilities to compare with from?
The test reads a .replies file from tests/qemucapabilitiesdata/,
parses it as if a QEMU binary had replied that way to our poking it,
figure out what the corresponding emulator capabilities are, format
them to XML and compare the result to the matching .xml file.
How should I proceed?
I've prepared patches that should address the issue you're seeing:
https://listman.redhat.com/archives/libvir-list/2022-August/233645.html
It'd be great if you could confirm that they work as intended.
--
Andrea Bolognani / Red Hat / Virtualization
648
days inactive
744
days old
10 comments
3 participants
participants (3)
-
Andrea Bolognani -
Christophe de Dinechin -
Daniel P. Berrangé