10:11 a.m.
Resend to libvir-list in case that is more appropriate:
Hi,
I get the following error when running libvirtd on MacOS as root:
2019-07-11 00:12:33.673+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error
: cannot set supplemental groups: Invalid argument
As a result `virsh capabilities` as root returns nothing:
$ sudo virsh capabilities | grep qemu
<baselabel type='qemu'>+0:+0</baselabel>
whereas running as a regular user works fine:
$ virsh capabilites | grep qemu
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
<domain type='qemu'/>
and by extension, running VMs as regular user works fine via
qemu:///session , but qemu:///system does not work.
It seems like setgroups is failing:
https://github.com/libvirt/libvirt/blob/v5.5.0/src/util/virutil.c#L1045-L...
Is this the expected behaviour?
Full output below:
$ sudo libvirtd
2019-07-11 00:12:33.379+0000: 123145573953536: info : libvirt version: 5.5.0
2019-07-11 00:12:33.379+0000: 123145573953536: warning :
virProcessGetStartTime:1070 : Process start time of pid 49746 not
available on this platform
2019-07-11 00:12:33.379+0000: 123145573953536: error :
virSysinfoReadDMI:1172 : internal error: Failed to find path for
dmidecode binary
2019-07-11 00:12:33.380+0000: 123145573953536: error :
virFileFindHugeTLBFS:3734 : this function is not supported by the
connection driver: virFileFindHugeTLBFS
2019-07-11 00:12:33.382+0000: 123145573953536: warning :
virQEMUCapsInit:919 : Failed to get host CPU cache info
2019-07-11 00:12:33.401+0000: 123145573953536: error :
virHostCPUGetTscInfo:1405 : Probing TSC is not supported on this
platform: Function not implemented
2019-07-11 00:12:33.401+0000: 123145573953536: error : virExec:521 :
Cannot find 'pm-is-supported' in path: No such file or directory
2019-07-11 00:12:33.401+0000: 123145573953536: warning :
virQEMUCapsInit:926 : Failed to get host power management capabilities
2019-07-11 00:12:33.401+0000: 123145573953536: error :
virNumaGetPages:988 : Operation not supported: page info is not
supported on this platform
2019-07-11 00:12:33.401+0000: 123145573953536: warning :
virQEMUCapsInit:933 : Failed to get pages info
2019-07-11 00:12:33.407+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-alpha for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.407+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-alpha: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-alpha for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.413+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-arm for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.413+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-arm: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-arm for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.419+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-arm for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.419+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-arm: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-arm for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.424+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-aarch64 for probing: libvirt: error
: cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.424+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-aarch64: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-aarch64 for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.431+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-cris for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.431+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-cris: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-cris for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.437+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-i386 for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.437+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-i386: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-i386 for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.443+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-lm32 for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.443+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-lm32: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-lm32 for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.449+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-m68k for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.449+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-m68k: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-m68k for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.497+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-microblaze for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.497+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-microblaze: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-microblaze for probing:
libvirt: error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.503+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-microblazeel for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.503+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-microblazeel: internal error: Failed to
start QEMU binary /usr/local/bin/qemu-system-microblazeel for probing:
libvirt: error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.509+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-mips for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.509+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-mips: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-mips for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.515+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-mipsel for probing: libvirt: error
: cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.515+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-mipsel: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-mipsel for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.521+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-mips64 for probing: libvirt: error
: cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.521+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-mips64: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-mips64 for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.527+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-mips64el for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.527+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-mips64el: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-mips64el for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.533+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-ppc for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.533+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-ppc: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-ppc for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.571+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-ppc64 for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.571+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-ppc64: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-ppc64 for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.576+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-ppc64 for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.576+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-ppc64: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-ppc64 for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.582+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-riscv32 for probing: libvirt: error
: cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.582+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-riscv32: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-riscv32 for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.588+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-riscv64 for probing: libvirt: error
: cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.588+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-riscv64: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-riscv64 for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.594+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-s390x for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.594+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-s390x: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-s390x for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.600+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-sh4 for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.600+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-sh4: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-sh4 for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.605+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-sh4eb for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.605+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-sh4eb: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-sh4eb for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.612+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-sparc for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.612+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-sparc: internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-sparc for probing: libvirt: error :
cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.619+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-sparc64 for probing: libvirt: error
: cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.619+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-sparc64: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-sparc64 for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.667+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-unicore32 for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.667+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-unicore32: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-unicore32 for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.673+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error
: cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.673+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-x86_64: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.679+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-xtensa for probing: libvirt: error
: cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.679+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-xtensa: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-xtensa for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.685+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-xtensaeb for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
2019-07-11 00:12:33.685+0000: 123145573953536: warning :
virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for
/usr/local/bin/qemu-system-xtensaeb: internal error: Failed to start
QEMU binary /usr/local/bin/qemu-system-xtensaeb for probing: libvirt:
error : cannot set supplemental groups: Invalid argument
--
Marcus Furlong
7:23 a.m.
On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
Resend to libvir-list in case that is more appropriate:
Hi,
I get the following error when running libvirtd on MacOS as root:
2019-07-11 00:12:33.673+0000: 123145573953536: error :
qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error
: cannot set supplemental groups: Invalid argument
Are you able to run 'strace' (or whatever MacOS eqiuv is) to see
the values passed to setgroups when it fails ?
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
11:47 a.m.
On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé <berrange(a)redhat.com> wrote:
On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
> Resend to libvir-list in case that is more appropriate:
>
>
> Hi,
>
> I get the following error when running libvirtd on MacOS as root:
>
> 2019-07-11 00:12:33.673+0000: 123145573953536: error :
> qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
> binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error
> : cannot set supplemental groups: Invalid argument
Are you able to run 'strace' (or whatever MacOS eqiuv is) to see
the values passed to setgroups when it fails ?
I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups
calls seem to be missing.
Looking at other sources, it seems like some have special treatment of
setgroups on MacOS, e.g. samba:
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ct...
Perhaps this is needed for libvirt?
Marcus.
--
Marcus Furlong
11:55 a.m.
On Wed, Aug 21, 2019 at 12:47:03PM -0400, Marcus Furlong wrote:
On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé
<berrange(a)redhat.com> wrote:
>
> On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
> > Resend to libvir-list in case that is more appropriate:
> >
> >
> > Hi,
> >
> > I get the following error when running libvirtd on MacOS as root:
> >
> > 2019-07-11 00:12:33.673+0000: 123145573953536: error :
> > qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
> > binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error
> > : cannot set supplemental groups: Invalid argument
>
> Are you able to run 'strace' (or whatever MacOS eqiuv is) to see
> the values passed to setgroups when it fails ?
I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups
calls seem to be missing.
Looking at other sources, it seems like some have special treatment of
setgroups on MacOS, e.g. samba:
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ct...
Perhaps this is needed for libvirt?
The capping of ngroups to NGROUPS_MAX looks like a possibe reason.
Adding this debug might show us if we're exceeding it:
diff --git a/src/util/virutil.c b/src/util/virutil.c
index 89d2cf011f..effc02b898 100644
--- a/src/util/virutil.c
+++ b/src/util/virutil.c
@@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups ATTRIBUTE_UNUSED,
}
# if HAVE_SETGROUPS
+ VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX);
if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) {
virReportSystemError(errno, "%s",
_("cannot set supplemental groups"));
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1:34 p.m.
On Wed, Aug 21, 2019 at 05:55:51PM +0100, Daniel P. Berrangé wrote:
On Wed, Aug 21, 2019 at 12:47:03PM -0400, Marcus Furlong wrote:
> On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé <berrange(a)redhat.com> wrote:
> >
> > On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
> > > Resend to libvir-list in case that is more appropriate:
> > >
> > >
> > > Hi,
> > >
> > > I get the following error when running libvirtd on MacOS as root:
> > >
> > > 2019-07-11 00:12:33.673+0000: 123145573953536: error :
> > > qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
> > > binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error
> > > : cannot set supplemental groups: Invalid argument
> >
> > Are you able to run 'strace' (or whatever MacOS eqiuv is) to see
> > the values passed to setgroups when it fails ?
>
> I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups
> calls seem to be missing.
>
> Looking at other sources, it seems like some have special treatment of
> setgroups on MacOS, e.g. samba:
>
>
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ct...
>
> Perhaps this is needed for libvirt?
The capping of ngroups to NGROUPS_MAX looks like a possibe reason.
Adding this debug might show us if we're exceeding it:
diff --git a/src/util/virutil.c b/src/util/virutil.c
index 89d2cf011f..effc02b898 100644
--- a/src/util/virutil.c
+++ b/src/util/virutil.c
@@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups ATTRIBUTE_UNUSED,
}
# if HAVE_SETGROUPS
+ VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX);
if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) {
virReportSystemError(errno, "%s",
_("cannot set supplemental groups"));
Yes, there's an overflow:
2019-08-21 18:25:37.943+0000: 123145413914624: debug : virSetUIDGID:1046 : setgroups 23
max 16
Related samba ticket (it also has references to the python and dovecot
issues):
https://bugzilla.samba.org/show_bug.cgi?id=8773
Thanks,
Roman
4:34 a.m.
On Wed, Aug 21, 2019 at 09:34:05PM +0300, Roman Bolshakov wrote:
On Wed, Aug 21, 2019 at 05:55:51PM +0100, Daniel P. Berrangé wrote:
> On Wed, Aug 21, 2019 at 12:47:03PM -0400, Marcus Furlong wrote:
> > On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé <berrange(a)redhat.com>
wrote:
> > >
> > > On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
> > > > Resend to libvir-list in case that is more appropriate:
> > > >
> > > >
> > > > Hi,
> > > >
> > > > I get the following error when running libvirtd on MacOS as root:
> > > >
> > > > 2019-07-11 00:12:33.673+0000: 123145573953536: error :
> > > > qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
> > > > binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt:
error
> > > > : cannot set supplemental groups: Invalid argument
> > >
> > > Are you able to run 'strace' (or whatever MacOS eqiuv is) to see
> > > the values passed to setgroups when it fails ?
> >
> > I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups
> > calls seem to be missing.
> >
> > Looking at other sources, it seems like some have special treatment of
> > setgroups on MacOS, e.g. samba:
> >
> >
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ct...
> >
> > Perhaps this is needed for libvirt?
>
> The capping of ngroups to NGROUPS_MAX looks like a possibe reason.
>
> Adding this debug might show us if we're exceeding it:
>
> diff --git a/src/util/virutil.c b/src/util/virutil.c
> index 89d2cf011f..effc02b898 100644
> --- a/src/util/virutil.c
> +++ b/src/util/virutil.c
> @@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups
ATTRIBUTE_UNUSED,
> }
>
> # if HAVE_SETGROUPS
> + VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX);
> if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) {
> virReportSystemError(errno, "%s",
> _("cannot set supplemental groups"));
>
>
Yes, there's an overflow:
2019-08-21 18:25:37.943+0000: 123145413914624: debug : virSetUIDGID:1046 : setgroups 23
max 16
Related samba ticket (it also has references to the python and dovecot
issues):
https://bugzilla.samba.org/show_bug.cgi?id=8773
The quick hack is to simply truncate groups to NGROUPS_MAX.
Reading that bug, the proper fix looks quite alot more complex
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
10:42 a.m.
On Thu, 22 Aug 2019 at 05:34, Daniel P. Berrangé <berrange(a)redhat.com> wrote:
On Wed, Aug 21, 2019 at 09:34:05PM +0300, Roman Bolshakov wrote:
> On Wed, Aug 21, 2019 at 05:55:51PM +0100, Daniel P. Berrangé wrote:
> > On Wed, Aug 21, 2019 at 12:47:03PM -0400, Marcus Furlong wrote:
> > > On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé
<berrange(a)redhat.com> wrote:
> > > >
> > > > On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
> > > > > Resend to libvir-list in case that is more appropriate:
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > I get the following error when running libvirtd on MacOS as
root:
> > > > >
> > > > > 2019-07-11 00:12:33.673+0000: 123145573953536: error :
> > > > > qemuProcessQMPLaunch:8501 : internal error: Failed to start
QEMU
> > > > > binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt:
error
> > > > > : cannot set supplemental groups: Invalid argument
> > > >
> > > > Are you able to run 'strace' (or whatever MacOS eqiuv is) to
see
> > > > the values passed to setgroups when it fails ?
> > >
> > > I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups
> > > calls seem to be missing.
> > >
> > > Looking at other sources, it seems like some have special treatment of
> > > setgroups on MacOS, e.g. samba:
> > >
> > >
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ct...
> > >
> > > Perhaps this is needed for libvirt?
> >
> > The capping of ngroups to NGROUPS_MAX looks like a possibe reason.
> >
> > Adding this debug might show us if we're exceeding it:
> >
> > diff --git a/src/util/virutil.c b/src/util/virutil.c
> > index 89d2cf011f..effc02b898 100644
> > --- a/src/util/virutil.c
> > +++ b/src/util/virutil.c
> > @@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups
ATTRIBUTE_UNUSED,
> > }
> >
> > # if HAVE_SETGROUPS
> > + VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX);
> > if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) {
> > virReportSystemError(errno, "%s",
> > _("cannot set supplemental groups"));
> >
> >
>
> Yes, there's an overflow:
> 2019-08-21 18:25:37.943+0000: 123145413914624: debug : virSetUIDGID:1046 : setgroups
23 max 16
>
> Related samba ticket (it also has references to the python and dovecot
> issues):
> https://bugzilla.samba.org/show_bug.cgi?id=8773
The quick hack is to simply truncate groups to NGROUPS_MAX.
Reading that bug, the proper fix looks quite alot more complex
Just to confirm that the root cause of the issue was the above block,
I bypassed the HAVE_SETGROUPS section completely on Darwin, and things
work as expected.
Where would be an appropriate place to track this bug?
Marcus.
--
Marcus Furlong
10:50 a.m.
On Fri, Aug 30, 2019 at 11:42:50AM -0400, Marcus Furlong wrote:
On Thu, 22 Aug 2019 at 05:34, Daniel P. Berrangé
<berrange(a)redhat.com> wrote:
>
> On Wed, Aug 21, 2019 at 09:34:05PM +0300, Roman Bolshakov wrote:
> > On Wed, Aug 21, 2019 at 05:55:51PM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Aug 21, 2019 at 12:47:03PM -0400, Marcus Furlong wrote:
> > > > On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé
<berrange(a)redhat.com> wrote:
> > > > >
> > > > > On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
> > > > > > Resend to libvir-list in case that is more appropriate:
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I get the following error when running libvirtd on MacOS as
root:
> > > > > >
> > > > > > 2019-07-11 00:12:33.673+0000: 123145573953536: error :
> > > > > > qemuProcessQMPLaunch:8501 : internal error: Failed to start
QEMU
> > > > > > binary /usr/local/bin/qemu-system-x86_64 for probing:
libvirt: error
> > > > > > : cannot set supplemental groups: Invalid argument
> > > > >
> > > > > Are you able to run 'strace' (or whatever MacOS eqiuv
is) to see
> > > > > the values passed to setgroups when it fails ?
> > > >
> > > > I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups
> > > > calls seem to be missing.
> > > >
> > > > Looking at other sources, it seems like some have special treatment
of
> > > > setgroups on MacOS, e.g. samba:
> > > >
> > > >
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ct...
> > > >
> > > > Perhaps this is needed for libvirt?
> > >
> > > The capping of ngroups to NGROUPS_MAX looks like a possibe reason.
> > >
> > > Adding this debug might show us if we're exceeding it:
> > >
> > > diff --git a/src/util/virutil.c b/src/util/virutil.c
> > > index 89d2cf011f..effc02b898 100644
> > > --- a/src/util/virutil.c
> > > +++ b/src/util/virutil.c
> > > @@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups
ATTRIBUTE_UNUSED,
> > > }
> > >
> > > # if HAVE_SETGROUPS
> > > + VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX);
> > > if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) {
> > > virReportSystemError(errno, "%s",
> > > _("cannot set supplemental
groups"));
> > >
> > >
> >
> > Yes, there's an overflow:
> > 2019-08-21 18:25:37.943+0000: 123145413914624: debug : virSetUIDGID:1046 :
setgroups 23 max 16
> >
> > Related samba ticket (it also has references to the python and dovecot
> > issues):
> > https://bugzilla.samba.org/show_bug.cgi?id=8773
>
> The quick hack is to simply truncate groups to NGROUPS_MAX.
>
> Reading that bug, the proper fix looks quite alot more complex
Just to confirm that the root cause of the issue was the above block,
I bypassed the HAVE_SETGROUPS section completely on Darwin, and things
work as expected.
Where would be an appropriate place to track this bug?
You can file a BZ here:
https://libvirt.org/bugs.html
I would note, however, that very few libvirt maintainers use or have
access to macOS. So if you're able to try creating a viable patch
yourself that would maximise likelihood of it getting fixed in a
reasonable amount of time.
I'll happily review & give feedback on the approach, but I'm not
able to actively test myself, besides a plain compile-check we
get via Travis CI.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
11:39 a.m.
[adding gnulib]
On 8/21/19 1:34 PM, Roman Bolshakov wrote:
>>>> I get the following error when running libvirtd on
MacOS as root:
>>>>
>>>> 2019-07-11 00:12:33.673+0000: 123145573953536: error :
>>>> qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU
>>>> binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error
>>>> : cannot set supplemental groups: Invalid argument
>>>
> +++ b/src/util/virutil.c
> @@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups
ATTRIBUTE_UNUSED,
> }
>
> # if HAVE_SETGROUPS
> + VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX);
> if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) {
> virReportSystemError(errno, "%s",
> _("cannot set supplemental groups"));
>
>
Yes, there's an overflow:
2019-08-21 18:25:37.943+0000: 123145413914624: debug : virSetUIDGID:1046 : setgroups 23
max 16
Related samba ticket (it also has references to the python and dovecot
issues):
https://bugzilla.samba.org/show_bug.cgi?id=8773
I wonder if gnulib could provide a workaround setgroups() that overcomes
this issue (it's better to maintain such a patch there, where it
benefits multiple programs, rather than just in libvirt).
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
8:33 p.m.
Hi Eric,
I wonder if gnulib could provide a workaround setgroups() that
overcomes
this issue
I don't see how a workaround could look like. The problem is not the value
of NGROUPS_MAX in user-space, but the same value NGROUPS_MAX in the kernel.
More precisely, in the Darwin kernel file bsd/kern/kern_prot.c there is a
function 'setgroups1', that contains the common implementation of the
setgroups() and initgroups() system call, and this function fails with EINVAL
if the number of groups in the set is > NGROUPS. In the kernel sources,
NGROUPS is defined as NGROUPS_MAX, and NGROUPS_MAX is defined as 16.
So, the situation on macOS has not changed since this page was written:
https://www.j3e.de/ngroups.html
What kind of workaround are you imagining? That we override open(),
access(), eaccess() to call setgroups() first, in an intelligent way?
That would be quite gross.
For what purpose is libvirt or QEMU using setgroups()?
Bruno
10:38 a.m.
On Fri, 30 Aug 2019 at 21:33, Bruno Haible <bruno(a)clisp.org> wrote:
Hi Eric,
> I wonder if gnulib could provide a workaround setgroups() that overcomes
> this issue
I don't see how a workaround could look like. The problem is not the value
of NGROUPS_MAX in user-space, but the same value NGROUPS_MAX in the kernel.
More precisely, in the Darwin kernel file bsd/kern/kern_prot.c there is a
function 'setgroups1', that contains the common implementation of the
setgroups() and initgroups() system call, and this function fails with EINVAL
if the number of groups in the set is > NGROUPS. In the kernel sources,
NGROUPS is defined as NGROUPS_MAX, and NGROUPS_MAX is defined as 16.
So, the situation on macOS has not changed since this page was written:
https://www.j3e.de/ngroups.html
What kind of workaround are you imagining? That we override open(),
access(), eaccess() to call setgroups() first, in an intelligent way?
That would be quite gross.
For what purpose is libvirt or QEMU using setgroups()?
FWIW I compiled libvirt without the setgroups code on Mac and it
worked as expected. Not sure what the implications of that are though?
Marcus.
--
Marcus Furlong
4:02 a.m.
On Sat, Sep 28, 2019 at 01:36:15PM +0200, Bruno Haible wrote:
Marcus Furlong wrote:
> FWIW I compiled libvirt without the setgroups code on Mac and it
> worked as expected. Not sure what the implications of that are though?
OK, then the fix would be to not use setgroups on Mac, and nothing to do
in gnulib. Right?
Not calling setgroups means the QEMU process doesn't run with any of
the supplementary groups associated with its user account, so this is
not really a working solution. It re-introduces the bug that the
setgroups call was added to fix.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
7:06 a.m.
Daniel P. Berrangé wrote:
> > FWIW I compiled libvirt without the setgroups code on Mac
and it
> > worked as expected. Not sure what the implications of that are though?
>
> OK, then the fix would be to not use setgroups on Mac, and nothing to do
> in gnulib. Right?
Not calling setgroups means the QEMU process doesn't run with any of
the supplementary groups associated with its user account, so this is
not really a working solution. It re-introduces the bug that the
setgroups call was added to fix.
For what purpose is libvirt or QEMU using setgroups()? What goes wrong if
setgroups() fails?
The problem is that the Darwin kernel does not support setting more than
NGROUPS_MAX (= 16) groups. So
- What happens when you have a user account which is in more than 16
groups? What do other processes do in this sitation?
- Is using the first 16 groups and ignoring the extra ones an acceptable
solution?
Bruno
7:16 a.m.
On Mon, Sep 30, 2019 at 02:06:07PM +0200, Bruno Haible wrote:
Daniel P. Berrangé wrote:
> > > FWIW I compiled libvirt without the setgroups code on Mac and it
> > > worked as expected. Not sure what the implications of that are though?
> >
> > OK, then the fix would be to not use setgroups on Mac, and nothing to do
> > in gnulib. Right?
>
> Not calling setgroups means the QEMU process doesn't run with any of
> the supplementary groups associated with its user account, so this is
> not really a working solution. It re-introduces the bug that the
> setgroups call was added to fix.
For what purpose is libvirt or QEMU using setgroups()? What goes wrong if
setgroups() fails?
QEMU potentially needs access to files owned by a supplementary group.
On Linux for example, /dev/kvm is often owned by 'kvm' group, but the
'qemu' user on Fedora has 'qemu' group as its primary group. So QEMU
would be unable to open /dev/kvm without the setgroups call to set up
supplementary groups.
The problem is that the Darwin kernel does not support setting more
than
NGROUPS_MAX (= 16) groups. So
- What happens when you have a user account which is in more than 16
groups? What do other processes do in this sitation?
Samba appears to use initgroups on Darwin, while clamping to 16 groups
only:
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ct...
- Is using the first 16 groups and ignoring the extra ones an
acceptable
solution?
Certainly that's better than just ignoring groups entirely, as it will
work for many more cases, even if not perfect.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
3:05 p.m.
Daniel P. Berrangé wrote:
> For what purpose is libvirt or QEMU using setgroups()? What goes
wrong if
> setgroups() fails?
QEMU potentially needs access to files owned by a supplementary group.
On Linux for example, /dev/kvm is often owned by 'kvm' group, but the
'qemu' user on Fedora has 'qemu' group as its primary group. So QEMU
would be unable to open /dev/kvm without the setgroups call to set up
supplementary groups.
Ah, it's libvirt which calls setgroups and qemu which needs the groups.
Then my suggested workaround that consists of overriding setgroups() and
open() won't work.
> - Is using the first 16 groups and ignoring the extra ones an
acceptable
> solution?
Certainly that's better than just ignoring groups entirely, as it will
work for many more cases, even if not perfect.
Hmm. If the group of /dev/kvm comes at 17th group, it will still not work.
I.e. it will be unreliable.
Then, how about if libvirt collects the set of groups that qemu might need
for accessing devices (surely less than 16), then fills up the remaining
up to 16 slots with secondary groups? Admittedly it makes qemu less
self-contained. But given that setgroups() works only for root on macOS [1]
I see no better way.
Bruno
[1]
https://developer.apple.com/library/archive/documentation/System/Conceptu...
11:07 a.m.
On Mon, 30 Sep 2019 at 21:05, Bruno Haible <bruno(a)clisp.org> wrote:
Daniel P. Berrangé wrote:
> > For what purpose is libvirt or QEMU using setgroups()? What goes wrong if
> > setgroups() fails?
On macOS, as far as I can see, everything works as expected without it.
So not sure if it's actually needed?
> QEMU potentially needs access to files owned by a supplementary
group.
> On Linux for example, /dev/kvm is often owned by 'kvm' group, but the
> 'qemu' user on Fedora has 'qemu' group as its primary group. So
QEMU
> would be unable to open /dev/kvm without the setgroups call to set up
> supplementary groups.
Ah, it's libvirt which calls setgroups and qemu which needs the groups.
Then my suggested workaround that consists of overriding setgroups() and
open() won't work.
> > - Is using the first 16 groups and ignoring the extra ones an acceptable
> > solution?
>
> Certainly that's better than just ignoring groups entirely, as it will
> work for many more cases, even if not perfect.
Hmm. If the group of /dev/kvm comes at 17th group, it will still not work.
I.e. it will be unreliable.
Then, how about if libvirt collects the set of groups that qemu might need
for accessing devices (surely less than 16), then fills up the remaining
up to 16 slots with secondary groups? Admittedly it makes qemu less
self-contained. But given that setgroups() works only for root on macOS [1]
I see no better way.
Note that /dev/kvm is for linux and does not exist on macOS.
Unless we identify specific devices on macOS that qemu requires access
to, then something like the following might work?
https://github.com/furlongm/libvirt/commit/01a1d3d0e37c7f81a04da2e9707ac1...
Seems to work correctly for me (virsh capabilities now returns the
correct output, and VMs run).
--
Marcus Furlong
1900
days inactive
1956
days old
16 comments
5 participants
participants (5)
-
Bruno Haible -
Daniel P. Berrangé -
Eric Blake -
Marcus Furlong -
Roman Bolshakov