[libvirt] Fwd: libvirtd failing on MacOS in setgroups
Resend to libvir-list in case that is more appropriate: Hi, I get the following error when running libvirtd on MacOS as root: 2019-07-11 00:12:33.673+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument As a result `virsh capabilities` as root returns nothing: $ sudo virsh capabilities | grep qemu <baselabel type='qemu'>+0:+0</baselabel> whereas running as a regular user works fine: $ virsh capabilites | grep qemu <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> <domain type='qemu'/> and by extension, running VMs as regular user works fine via qemu:///session , but qemu:///system does not work. It seems like setgroups is failing: https://github.com/libvirt/libvirt/blob/v5.5.0/src/util/virutil.c#L1045-L105... Is this the expected behaviour? Full output below: $ sudo libvirtd 2019-07-11 00:12:33.379+0000: 123145573953536: info : libvirt version: 5.5.0 2019-07-11 00:12:33.379+0000: 123145573953536: warning : virProcessGetStartTime:1070 : Process start time of pid 49746 not available on this platform 2019-07-11 00:12:33.379+0000: 123145573953536: error : virSysinfoReadDMI:1172 : internal error: Failed to find path for dmidecode binary 2019-07-11 00:12:33.380+0000: 123145573953536: error : virFileFindHugeTLBFS:3734 : this function is not supported by the connection driver: virFileFindHugeTLBFS 2019-07-11 00:12:33.382+0000: 123145573953536: warning : virQEMUCapsInit:919 : Failed to get host CPU cache info 2019-07-11 00:12:33.401+0000: 123145573953536: error : virHostCPUGetTscInfo:1405 : Probing TSC is not supported on this platform: Function not implemented 2019-07-11 00:12:33.401+0000: 123145573953536: error : virExec:521 : Cannot find 'pm-is-supported' in path: No such file or directory 2019-07-11 00:12:33.401+0000: 123145573953536: warning : virQEMUCapsInit:926 : Failed to get host power management capabilities 2019-07-11 00:12:33.401+0000: 123145573953536: error : virNumaGetPages:988 : Operation not supported: page info is not supported on this platform 2019-07-11 00:12:33.401+0000: 123145573953536: warning : virQEMUCapsInit:933 : Failed to get pages info 2019-07-11 00:12:33.407+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-alpha for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.407+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-alpha: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-alpha for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.413+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-arm for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.413+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-arm: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-arm for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.419+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-arm for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.419+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-arm: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-arm for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.424+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-aarch64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.424+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-aarch64: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-aarch64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.431+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-cris for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.431+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-cris: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-cris for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.437+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-i386 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.437+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-i386: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-i386 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.443+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-lm32 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.443+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-lm32: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-lm32 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.449+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-m68k for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.449+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-m68k: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-m68k for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.497+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-microblaze for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.497+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-microblaze: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-microblaze for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.503+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-microblazeel for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.503+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-microblazeel: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-microblazeel for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.509+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-mips for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.509+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-mips: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-mips for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.515+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-mipsel for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.515+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-mipsel: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-mipsel for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.521+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-mips64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.521+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-mips64: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-mips64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.527+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-mips64el for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.527+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-mips64el: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-mips64el for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.533+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-ppc for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.533+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-ppc: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-ppc for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.571+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-ppc64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.571+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-ppc64: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-ppc64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.576+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-ppc64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.576+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-ppc64: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-ppc64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.582+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-riscv32 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.582+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-riscv32: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-riscv32 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.588+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-riscv64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.588+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-riscv64: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-riscv64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.594+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-s390x for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.594+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-s390x: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-s390x for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.600+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-sh4 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.600+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-sh4: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-sh4 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.605+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-sh4eb for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.605+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-sh4eb: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-sh4eb for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.612+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-sparc for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.612+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-sparc: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-sparc for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.619+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-sparc64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.619+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-sparc64: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-sparc64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.667+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-unicore32 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.667+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-unicore32: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-unicore32 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.673+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.673+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-x86_64: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.679+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-xtensa for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.679+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-xtensa: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-xtensa for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.685+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-xtensaeb for probing: libvirt: error : cannot set supplemental groups: Invalid argument 2019-07-11 00:12:33.685+0000: 123145573953536: warning : virQEMUCapsLogProbeFailure:4578 : Failed to probe capabilities for /usr/local/bin/qemu-system-xtensaeb: internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-xtensaeb for probing: libvirt: error : cannot set supplemental groups: Invalid argument -- Marcus Furlong
On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
Resend to libvir-list in case that is more appropriate:
Hi,
I get the following error when running libvirtd on MacOS as root:
2019-07-11 00:12:33.673+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument
Are you able to run 'strace' (or whatever MacOS eqiuv is) to see the values passed to setgroups when it fails ? Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
Resend to libvir-list in case that is more appropriate:
Hi,
I get the following error when running libvirtd on MacOS as root:
2019-07-11 00:12:33.673+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument
Are you able to run 'strace' (or whatever MacOS eqiuv is) to see the values passed to setgroups when it fails ?
I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups calls seem to be missing. Looking at other sources, it seems like some have special treatment of setgroups on MacOS, e.g. samba: https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ctx.c... Perhaps this is needed for libvirt? Marcus. -- Marcus Furlong
On Wed, Aug 21, 2019 at 12:47:03PM -0400, Marcus Furlong wrote:
On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
Resend to libvir-list in case that is more appropriate:
Hi,
I get the following error when running libvirtd on MacOS as root:
2019-07-11 00:12:33.673+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument
Are you able to run 'strace' (or whatever MacOS eqiuv is) to see the values passed to setgroups when it fails ?
I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups calls seem to be missing.
Looking at other sources, it seems like some have special treatment of setgroups on MacOS, e.g. samba:
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ctx.c...
Perhaps this is needed for libvirt?
The capping of ngroups to NGROUPS_MAX looks like a possibe reason. Adding this debug might show us if we're exceeding it: diff --git a/src/util/virutil.c b/src/util/virutil.c index 89d2cf011f..effc02b898 100644 --- a/src/util/virutil.c +++ b/src/util/virutil.c @@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups ATTRIBUTE_UNUSED, } # if HAVE_SETGROUPS + VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX); if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) { virReportSystemError(errno, "%s", _("cannot set supplemental groups")); Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Aug 21, 2019 at 05:55:51PM +0100, Daniel P. Berrangé wrote:
On Wed, Aug 21, 2019 at 12:47:03PM -0400, Marcus Furlong wrote:
On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
Resend to libvir-list in case that is more appropriate:
Hi,
I get the following error when running libvirtd on MacOS as root:
2019-07-11 00:12:33.673+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument
Are you able to run 'strace' (or whatever MacOS eqiuv is) to see the values passed to setgroups when it fails ?
I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups calls seem to be missing.
Looking at other sources, it seems like some have special treatment of setgroups on MacOS, e.g. samba:
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ctx.c...
Perhaps this is needed for libvirt?
The capping of ngroups to NGROUPS_MAX looks like a possibe reason.
Adding this debug might show us if we're exceeding it:
diff --git a/src/util/virutil.c b/src/util/virutil.c index 89d2cf011f..effc02b898 100644 --- a/src/util/virutil.c +++ b/src/util/virutil.c @@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups ATTRIBUTE_UNUSED, }
# if HAVE_SETGROUPS + VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX); if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) { virReportSystemError(errno, "%s", _("cannot set supplemental groups"));
Yes, there's an overflow: 2019-08-21 18:25:37.943+0000: 123145413914624: debug : virSetUIDGID:1046 : setgroups 23 max 16 Related samba ticket (it also has references to the python and dovecot issues): https://bugzilla.samba.org/show_bug.cgi?id=8773 Thanks, Roman
On Wed, Aug 21, 2019 at 09:34:05PM +0300, Roman Bolshakov wrote:
On Wed, Aug 21, 2019 at 05:55:51PM +0100, Daniel P. Berrangé wrote:
On Wed, Aug 21, 2019 at 12:47:03PM -0400, Marcus Furlong wrote:
On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
Resend to libvir-list in case that is more appropriate:
Hi,
I get the following error when running libvirtd on MacOS as root:
2019-07-11 00:12:33.673+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument
Are you able to run 'strace' (or whatever MacOS eqiuv is) to see the values passed to setgroups when it fails ?
I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups calls seem to be missing.
Looking at other sources, it seems like some have special treatment of setgroups on MacOS, e.g. samba:
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ctx.c...
Perhaps this is needed for libvirt?
The capping of ngroups to NGROUPS_MAX looks like a possibe reason.
Adding this debug might show us if we're exceeding it:
diff --git a/src/util/virutil.c b/src/util/virutil.c index 89d2cf011f..effc02b898 100644 --- a/src/util/virutil.c +++ b/src/util/virutil.c @@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups ATTRIBUTE_UNUSED, }
# if HAVE_SETGROUPS + VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX); if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) { virReportSystemError(errno, "%s", _("cannot set supplemental groups"));
Yes, there's an overflow: 2019-08-21 18:25:37.943+0000: 123145413914624: debug : virSetUIDGID:1046 : setgroups 23 max 16
Related samba ticket (it also has references to the python and dovecot issues): https://bugzilla.samba.org/show_bug.cgi?id=8773
The quick hack is to simply truncate groups to NGROUPS_MAX. Reading that bug, the proper fix looks quite alot more complex Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, 22 Aug 2019 at 05:34, Daniel P. Berrangé <berrange@redhat.com> wrote:
On Wed, Aug 21, 2019 at 09:34:05PM +0300, Roman Bolshakov wrote:
On Wed, Aug 21, 2019 at 05:55:51PM +0100, Daniel P. Berrangé wrote:
On Wed, Aug 21, 2019 at 12:47:03PM -0400, Marcus Furlong wrote:
On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote:
Resend to libvir-list in case that is more appropriate:
Hi,
I get the following error when running libvirtd on MacOS as root:
2019-07-11 00:12:33.673+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument
Are you able to run 'strace' (or whatever MacOS eqiuv is) to see the values passed to setgroups when it fails ?
I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups calls seem to be missing.
Looking at other sources, it seems like some have special treatment of setgroups on MacOS, e.g. samba:
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ctx.c...
Perhaps this is needed for libvirt?
The capping of ngroups to NGROUPS_MAX looks like a possibe reason.
Adding this debug might show us if we're exceeding it:
diff --git a/src/util/virutil.c b/src/util/virutil.c index 89d2cf011f..effc02b898 100644 --- a/src/util/virutil.c +++ b/src/util/virutil.c @@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups ATTRIBUTE_UNUSED, }
# if HAVE_SETGROUPS + VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX); if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) { virReportSystemError(errno, "%s", _("cannot set supplemental groups"));
Yes, there's an overflow: 2019-08-21 18:25:37.943+0000: 123145413914624: debug : virSetUIDGID:1046 : setgroups 23 max 16
Related samba ticket (it also has references to the python and dovecot issues): https://bugzilla.samba.org/show_bug.cgi?id=8773
The quick hack is to simply truncate groups to NGROUPS_MAX.
Reading that bug, the proper fix looks quite alot more complex
Just to confirm that the root cause of the issue was the above block, I bypassed the HAVE_SETGROUPS section completely on Darwin, and things work as expected. Where would be an appropriate place to track this bug? Marcus. -- Marcus Furlong
On Fri, Aug 30, 2019 at 11:42:50AM -0400, Marcus Furlong wrote:
On Thu, 22 Aug 2019 at 05:34, Daniel P. Berrangé <berrange@redhat.com> wrote:
On Wed, Aug 21, 2019 at 09:34:05PM +0300, Roman Bolshakov wrote:
On Wed, Aug 21, 2019 at 05:55:51PM +0100, Daniel P. Berrangé wrote:
On Wed, Aug 21, 2019 at 12:47:03PM -0400, Marcus Furlong wrote:
On Wed, 21 Aug 2019 at 08:23, Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Aug 20, 2019 at 11:11:07AM -0400, Marcus Furlong wrote: > Resend to libvir-list in case that is more appropriate: > > > Hi, > > I get the following error when running libvirtd on MacOS as root: > > 2019-07-11 00:12:33.673+0000: 123145573953536: error : > qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU > binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error > : cannot set supplemental groups: Invalid argument
Are you able to run 'strace' (or whatever MacOS eqiuv is) to see the values passed to setgroups when it fails ?
I ran `dtruss -f -l -s /usr/local/sbin/libvirtd` but the setgroups calls seem to be missing.
Looking at other sources, it seems like some have special treatment of setgroups on MacOS, e.g. samba:
https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ctx.c...
Perhaps this is needed for libvirt?
The capping of ngroups to NGROUPS_MAX looks like a possibe reason.
Adding this debug might show us if we're exceeding it:
diff --git a/src/util/virutil.c b/src/util/virutil.c index 89d2cf011f..effc02b898 100644 --- a/src/util/virutil.c +++ b/src/util/virutil.c @@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups ATTRIBUTE_UNUSED, }
# if HAVE_SETGROUPS + VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX); if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) { virReportSystemError(errno, "%s", _("cannot set supplemental groups"));
Yes, there's an overflow: 2019-08-21 18:25:37.943+0000: 123145413914624: debug : virSetUIDGID:1046 : setgroups 23 max 16
Related samba ticket (it also has references to the python and dovecot issues): https://bugzilla.samba.org/show_bug.cgi?id=8773
The quick hack is to simply truncate groups to NGROUPS_MAX.
Reading that bug, the proper fix looks quite alot more complex
Just to confirm that the root cause of the issue was the above block, I bypassed the HAVE_SETGROUPS section completely on Darwin, and things work as expected.
Where would be an appropriate place to track this bug?
You can file a BZ here: https://libvirt.org/bugs.html I would note, however, that very few libvirt maintainers use or have access to macOS. So if you're able to try creating a viable patch yourself that would maximise likelihood of it getting fixed in a reasonable amount of time. I'll happily review & give feedback on the approach, but I'm not able to actively test myself, besides a plain compile-check we get via Travis CI. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
[adding gnulib] On 8/21/19 1:34 PM, Roman Bolshakov wrote:
I get the following error when running libvirtd on MacOS as root:
2019-07-11 00:12:33.673+0000: 123145573953536: error : qemuProcessQMPLaunch:8501 : internal error: Failed to start QEMU binary /usr/local/bin/qemu-system-x86_64 for probing: libvirt: error : cannot set supplemental groups: Invalid argument
+++ b/src/util/virutil.c @@ -1043,6 +1043,7 @@ virSetUIDGID(uid_t uid, gid_t gid, gid_t *groups ATTRIBUTE_UNUSED, }
# if HAVE_SETGROUPS + VIR_DEBUG("setgroups %d max %d", ngroups, NGROUPS_MAX); if (gid != (gid_t)-1 && setgroups(ngroups, groups) < 0) { virReportSystemError(errno, "%s", _("cannot set supplemental groups"));
Yes, there's an overflow: 2019-08-21 18:25:37.943+0000: 123145413914624: debug : virSetUIDGID:1046 : setgroups 23 max 16
Related samba ticket (it also has references to the python and dovecot issues): https://bugzilla.samba.org/show_bug.cgi?id=8773
I wonder if gnulib could provide a workaround setgroups() that overcomes this issue (it's better to maintain such a patch there, where it benefits multiple programs, rather than just in libvirt). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Hi Eric,
I wonder if gnulib could provide a workaround setgroups() that overcomes this issue
I don't see how a workaround could look like. The problem is not the value of NGROUPS_MAX in user-space, but the same value NGROUPS_MAX in the kernel. More precisely, in the Darwin kernel file bsd/kern/kern_prot.c there is a function 'setgroups1', that contains the common implementation of the setgroups() and initgroups() system call, and this function fails with EINVAL if the number of groups in the set is > NGROUPS. In the kernel sources, NGROUPS is defined as NGROUPS_MAX, and NGROUPS_MAX is defined as 16. So, the situation on macOS has not changed since this page was written: https://www.j3e.de/ngroups.html What kind of workaround are you imagining? That we override open(), access(), eaccess() to call setgroups() first, in an intelligent way? That would be quite gross. For what purpose is libvirt or QEMU using setgroups()? Bruno
On Fri, 30 Aug 2019 at 21:33, Bruno Haible <bruno@clisp.org> wrote:
Hi Eric,
I wonder if gnulib could provide a workaround setgroups() that overcomes this issue
I don't see how a workaround could look like. The problem is not the value of NGROUPS_MAX in user-space, but the same value NGROUPS_MAX in the kernel. More precisely, in the Darwin kernel file bsd/kern/kern_prot.c there is a function 'setgroups1', that contains the common implementation of the setgroups() and initgroups() system call, and this function fails with EINVAL if the number of groups in the set is > NGROUPS. In the kernel sources, NGROUPS is defined as NGROUPS_MAX, and NGROUPS_MAX is defined as 16.
So, the situation on macOS has not changed since this page was written: https://www.j3e.de/ngroups.html
What kind of workaround are you imagining? That we override open(), access(), eaccess() to call setgroups() first, in an intelligent way? That would be quite gross.
For what purpose is libvirt or QEMU using setgroups()?
FWIW I compiled libvirt without the setgroups code on Mac and it worked as expected. Not sure what the implications of that are though? Marcus. -- Marcus Furlong
On Sat, Sep 28, 2019 at 01:36:15PM +0200, Bruno Haible wrote:
Marcus Furlong wrote:
FWIW I compiled libvirt without the setgroups code on Mac and it worked as expected. Not sure what the implications of that are though?
OK, then the fix would be to not use setgroups on Mac, and nothing to do in gnulib. Right?
Not calling setgroups means the QEMU process doesn't run with any of the supplementary groups associated with its user account, so this is not really a working solution. It re-introduces the bug that the setgroups call was added to fix. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Daniel P. Berrangé wrote:
FWIW I compiled libvirt without the setgroups code on Mac and it worked as expected. Not sure what the implications of that are though?
OK, then the fix would be to not use setgroups on Mac, and nothing to do in gnulib. Right?
Not calling setgroups means the QEMU process doesn't run with any of the supplementary groups associated with its user account, so this is not really a working solution. It re-introduces the bug that the setgroups call was added to fix.
For what purpose is libvirt or QEMU using setgroups()? What goes wrong if setgroups() fails? The problem is that the Darwin kernel does not support setting more than NGROUPS_MAX (= 16) groups. So - What happens when you have a user account which is in more than 16 groups? What do other processes do in this sitation? - Is using the first 16 groups and ignoring the extra ones an acceptable solution? Bruno
On Mon, Sep 30, 2019 at 02:06:07PM +0200, Bruno Haible wrote:
Daniel P. Berrangé wrote:
FWIW I compiled libvirt without the setgroups code on Mac and it worked as expected. Not sure what the implications of that are though?
OK, then the fix would be to not use setgroups on Mac, and nothing to do in gnulib. Right?
Not calling setgroups means the QEMU process doesn't run with any of the supplementary groups associated with its user account, so this is not really a working solution. It re-introduces the bug that the setgroups call was added to fix.
For what purpose is libvirt or QEMU using setgroups()? What goes wrong if setgroups() fails?
QEMU potentially needs access to files owned by a supplementary group. On Linux for example, /dev/kvm is often owned by 'kvm' group, but the 'qemu' user on Fedora has 'qemu' group as its primary group. So QEMU would be unable to open /dev/kvm without the setgroups call to set up supplementary groups.
The problem is that the Darwin kernel does not support setting more than NGROUPS_MAX (= 16) groups. So - What happens when you have a user account which is in more than 16 groups? What do other processes do in this sitation?
Samba appears to use initgroups on Darwin, while clamping to 16 groups only: https://github.com/samba-team/samba/blob/v4-11-stable/source3/smbd/sec_ctx.c...
- Is using the first 16 groups and ignoring the extra ones an acceptable solution?
Certainly that's better than just ignoring groups entirely, as it will work for many more cases, even if not perfect. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Daniel P. Berrangé wrote:
For what purpose is libvirt or QEMU using setgroups()? What goes wrong if setgroups() fails?
QEMU potentially needs access to files owned by a supplementary group. On Linux for example, /dev/kvm is often owned by 'kvm' group, but the 'qemu' user on Fedora has 'qemu' group as its primary group. So QEMU would be unable to open /dev/kvm without the setgroups call to set up supplementary groups.
Ah, it's libvirt which calls setgroups and qemu which needs the groups. Then my suggested workaround that consists of overriding setgroups() and open() won't work.
- Is using the first 16 groups and ignoring the extra ones an acceptable solution?
Certainly that's better than just ignoring groups entirely, as it will work for many more cases, even if not perfect.
Hmm. If the group of /dev/kvm comes at 17th group, it will still not work. I.e. it will be unreliable. Then, how about if libvirt collects the set of groups that qemu might need for accessing devices (surely less than 16), then fills up the remaining up to 16 slots with secondary groups? Admittedly it makes qemu less self-contained. But given that setgroups() works only for root on macOS [1] I see no better way. Bruno [1] https://developer.apple.com/library/archive/documentation/System/Conceptual/...
On Mon, 30 Sep 2019 at 21:05, Bruno Haible <bruno@clisp.org> wrote:
Daniel P. Berrangé wrote:
For what purpose is libvirt or QEMU using setgroups()? What goes wrong if setgroups() fails?
On macOS, as far as I can see, everything works as expected without it. So not sure if it's actually needed?
QEMU potentially needs access to files owned by a supplementary group. On Linux for example, /dev/kvm is often owned by 'kvm' group, but the 'qemu' user on Fedora has 'qemu' group as its primary group. So QEMU would be unable to open /dev/kvm without the setgroups call to set up supplementary groups.
Ah, it's libvirt which calls setgroups and qemu which needs the groups. Then my suggested workaround that consists of overriding setgroups() and open() won't work.
- Is using the first 16 groups and ignoring the extra ones an acceptable solution?
Certainly that's better than just ignoring groups entirely, as it will work for many more cases, even if not perfect.
Hmm. If the group of /dev/kvm comes at 17th group, it will still not work. I.e. it will be unreliable.
Then, how about if libvirt collects the set of groups that qemu might need for accessing devices (surely less than 16), then fills up the remaining up to 16 slots with secondary groups? Admittedly it makes qemu less self-contained. But given that setgroups() works only for root on macOS [1] I see no better way.
Note that /dev/kvm is for linux and does not exist on macOS. Unless we identify specific devices on macOS that qemu requires access to, then something like the following might work? https://github.com/furlongm/libvirt/commit/01a1d3d0e37c7f81a04da2e9707ac1c39... Seems to work correctly for me (virsh capabilities now returns the correct output, and VMs run). -- Marcus Furlong
participants (5)
-
Bruno Haible -
Daniel P. Berrangé -
Eric Blake -
Marcus Furlong -
Roman Bolshakov