[libvirt PATCH v7 0/3] Ignore EPERM on implicit clearing of VF VLAN ID
SmartNIC DPUs may not expose some privileged eswitch operations to the hypervisor hosts. For example, this happens with Bluefield devices running in the ECPF (default) mode [1] for security reasons. While VF MAC address programming is possible via an RTM_SETLINK operation, trying to set a VLAN ID in the same operation will fail with EPERM. In the kernel a relevant call chain may look like do_setlink -> do_setvfinfo -> dev->netdev_ops->set_vf_vlan which calls a driver-specific function like [2] eventually. The equivalent ip link commands below provide an illustration: 1. This works: sudo ip link set enp130s0f0 vf 2 mac de:ad:be:ef:ca:fe 2. Setting (or clearing) a VLAN fails with EPERM: sudo ip link set enp130s0f0 vf 2 vlan 0 RTNETLINK answers: Operation not permitted 3. This is what Libvirt attempts to do today (when trying to clear a VF VLAN at the same time as programming a VF MAC). sudo ip link set enp130s0f0 vf 2 vlan 0 mac de:ad:be:ef:ca:fe RTNETLINK answers: Operation not permitted If setting an explicit VLAN ID results in an EPERM, clearing a VLAN (setting a VLAN ID to 0) can be handled gracefully by ignoring the EPERM error with the rationale being that if we cannot set this state in the first place, we cannot clear it either. Thus, virNetDevSetVfConfig is split into two distinct functions. If clearing a VLAN ID fails with EPERM when clearing is implicit, the error is simply ignored. For explicit clearing EPERM is still a fatal error. Both new functions rely virNetDevSendVfSetLinkRequest that implements common functionality related to formatting a request, sending it and handling error conditions and returns 0 or an error since in both cases the payload is either NLMSG_DONE (no error) or NLMSG_ERROR where an error message is needed by the caller to handle known cases appropriately. This function allows the conditional code to be unit tested. An alternative to this could be providing a higher level control plane mechanism that would provide metadata about a device being remotely managed in which case Libvirt would avoid trying to set or clear a VLAN ID. This would be more complicated since other software (like Nova in the OpenStack case) would have to annotate every guest device with an attribute indicating whether a device is remotely managed or not based on operator provided configuration so that Libvirt can act on this and avoid VLAN programming. https://gitlab.com/dmitriis/libvirt/-/pipelines/420603489 v4 changes: * Split the change into several patches; * Added a missing test case; * Reworked error handling; * Added a way to pass a NULL vlan id to preserve the old behavior when needed. v5 changes: * added the news file entry; * fixed vlan handling in virNetDevSetVfVlan (for an issue introduced in v4). v6 changes: Refactoring, addressed: https://listman.redhat.com/archives/libvir-list/2021-November/msg00521.html v7 change: * Rebased on top of the latest changes; * Added stubs for platforms without libnl (fixed Windows builds). [1] https://docs.mellanox.com/display/BlueFieldSWv35111601/Modes+of+Operation#Mo... [2] https://github.com/torvalds/linux/blob/v5.15/drivers/net/ethernet/mellanox/m... Dmitrii Shcherbakov (3): Set VF MAC and VLAN ID in two different operations Allow VF vlanid to be passed as a pointer Ignore EPERM on implicit clearing of VF VLAN ID NEWS.rst | 14 ++ src/hypervisor/virhostdev.c | 4 +- src/libvirt_private.syms | 7 + src/util/virnetdev.c | 256 +++++++++++++++++++++++++----------- src/util/virnetdevpriv.h | 44 +++++++ tests/virnetdevtest.c | 249 ++++++++++++++++++++++++++++++++++- 6 files changed, 496 insertions(+), 78 deletions(-) create mode 100644 src/util/virnetdevpriv.h -- 2.32.0
This has a benefit of being able to handle error codes for those operations separately which is useful when drivers allow setting a MAC address but do not allow setting a VLAN (which is the case with some SmartNIC DPUs). Signed-off-by: Dmitrii Shcherbakov <dmitrii.shcherbakov@canonical.com> --- src/libvirt_private.syms | 7 ++ src/util/virnetdev.c | 226 ++++++++++++++++++++++++++------------ src/util/virnetdevpriv.h | 44 ++++++++ tests/virnetdevtest.c | 230 ++++++++++++++++++++++++++++++++++++++- 4 files changed, 434 insertions(+), 73 deletions(-) create mode 100644 src/util/virnetdevpriv.h diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index 7be5b51100..fa7908625c 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -2851,6 +2851,13 @@ virNetDevOpenvswitchSetTimeout; virNetDevOpenvswitchUpdateVlan; +# util/virnetdevpriv.h +virNetDevSendVfSetLinkRequest; +virNetDevSetVfConfig; +virNetDevSetVfMac; +virNetDevSetVfVlan; + + # util/virnetdevtap.h virNetDevTapAttachBridge; virNetDevTapCreate; diff --git a/src/util/virnetdev.c b/src/util/virnetdev.c index 58f7360a0f..ea8cae0dcf 100644 --- a/src/util/virnetdev.c +++ b/src/util/virnetdev.c @@ -19,7 +19,9 @@ #include <config.h> #include <math.h> -#include "virnetdev.h" +#define LIBVIRT_VIRNETDEVPRIV_H_ALLOW + +#include "virnetdevpriv.h" #include "viralloc.h" #include "virnetlink.h" #include "virmacaddr.h" @@ -1527,16 +1529,12 @@ static struct nla_policy ifla_vfstats_policy[IFLA_VF_STATS_MAX+1] = { [IFLA_VF_STATS_MULTICAST] = { .type = NLA_U64 }, }; - -static int -virNetDevSetVfConfig(const char *ifname, int vf, - const virMacAddr *macaddr, int vlanid, - bool *allowRetry) +int +virNetDevSendVfSetLinkRequest(const char *ifname, int vfInfoType, + const void *payload, const size_t payloadLen) { - int rc = -1; - char macstr[VIR_MAC_STRING_BUFLEN]; g_autofree struct nlmsghdr *resp = NULL; - struct nlmsgerr *err; + struct nlmsgerr *err = NULL; unsigned int recvbuflen = 0; struct nl_msg *nl_msg; struct nlattr *vfinfolist, *vfinfo; @@ -1544,9 +1542,11 @@ virNetDevSetVfConfig(const char *ifname, int vf, .ifi_family = AF_UNSPEC, .ifi_index = -1, }; + int rc = 0; - if (!macaddr && vlanid < 0) + if (payload == NULL || payloadLen == 0) { return -1; + } nl_msg = virNetlinkMsgNew(RTM_SETLINK, NLM_F_REQUEST); @@ -1564,30 +1564,8 @@ virNetDevSetVfConfig(const char *ifname, int vf, if (!(vfinfo = nla_nest_start(nl_msg, IFLA_VF_INFO))) goto buffer_too_small; - if (macaddr) { - struct ifla_vf_mac ifla_vf_mac = { - .vf = vf, - .mac = { 0, }, - }; - - virMacAddrGetRaw(macaddr, ifla_vf_mac.mac); - - if (nla_put(nl_msg, IFLA_VF_MAC, sizeof(ifla_vf_mac), - &ifla_vf_mac) < 0) - goto buffer_too_small; - } - - if (vlanid >= 0) { - struct ifla_vf_vlan ifla_vf_vlan = { - .vf = vf, - .vlan = vlanid, - .qos = 0, - }; - - if (nla_put(nl_msg, IFLA_VF_VLAN, sizeof(ifla_vf_vlan), - &ifla_vf_vlan) < 0) - goto buffer_too_small; - } + if (nla_put(nl_msg, vfInfoType, payloadLen, payload) < 0) + goto buffer_too_small; nla_nest_end(nl_msg, vfinfo); nla_nest_end(nl_msg, vfinfolist); @@ -1600,48 +1578,20 @@ virNetDevSetVfConfig(const char *ifname, int vf, goto malformed_resp; switch (resp->nlmsg_type) { - case NLMSG_ERROR: - err = (struct nlmsgerr *)NLMSG_DATA(resp); - if (resp->nlmsg_len < NLMSG_LENGTH(sizeof(*err))) + case NLMSG_ERROR: + err = (struct nlmsgerr *)NLMSG_DATA(resp); + if (resp->nlmsg_len < NLMSG_LENGTH(sizeof(*err))) + goto malformed_resp; + rc = err->error; + break; + case NLMSG_DONE: + rc = 0; + break; + default: goto malformed_resp; - - /* if allowRetry is true and the error was EINVAL, then - * silently return a failure so the caller can retry with a - * different MAC address - */ - if (err->error == -EINVAL && *allowRetry && - macaddr && !virMacAddrCmp(macaddr, &zeroMAC)) { - goto cleanup; - } else if (err->error) { - /* other errors are permanent */ - virReportSystemError(-err->error, - _("Cannot set interface MAC/vlanid to %s/%d " - "for ifname %s vf %d"), - (macaddr - ? virMacAddrFormat(macaddr, macstr) - : "(unchanged)"), - vlanid, - ifname ? ifname : "(unspecified)", - vf); - *allowRetry = false; /* no use retrying */ - goto cleanup; - } - break; - - case NLMSG_DONE: - break; - - default: - goto malformed_resp; } - rc = 0; cleanup: - VIR_DEBUG("RTM_SETLINK %s vf %d MAC=%s vlanid=%d - %s", - ifname, vf, - macaddr ? virMacAddrFormat(macaddr, macstr) : "(unchanged)", - vlanid, rc < 0 ? "Fail" : "Success"); - nlmsg_free(nl_msg); return rc; @@ -1656,6 +1606,100 @@ virNetDevSetVfConfig(const char *ifname, int vf, goto cleanup; } +int +virNetDevSetVfVlan(const char *ifname, int vf, int vlanid) +{ + int rc = 0; + int requestError = 0; + + struct ifla_vf_vlan ifla_vf_vlan = { + .vf = vf, + .vlan = vlanid, + .qos = 0, + }; + + /* VLAN ids 0 and 4095 are reserved per 802.1Q but are valid values. */ + if ((vlanid < 0 || vlanid > 4095)) { + return -ERANGE; + } + + requestError = virNetDevSendVfSetLinkRequest(ifname, IFLA_VF_VLAN, + &ifla_vf_vlan, sizeof(ifla_vf_vlan)); + + if (requestError) { + virReportSystemError(-requestError, + _("Cannot set interface vlanid to %d " + "for ifname %s vf %d"), + vlanid, ifname ? ifname : "(unspecified)", vf); + rc = requestError; + } + VIR_DEBUG("RTM_SETLINK %s vf %d vlanid=%d - %s", + ifname, vf, + vlanid, rc < 0 ? "Fail" : "Success"); + return rc; +} + +int +virNetDevSetVfMac(const char *ifname, int vf, + const virMacAddr *macaddr, + bool *allowRetry) +{ + int rc = 0; + int requestError = 0; + char macstr[VIR_MAC_STRING_BUFLEN]; + + struct ifla_vf_mac ifla_vf_mac = { + .vf = vf, + .mac = { 0, }, + }; + + if (macaddr == NULL || allowRetry == NULL) + return -EINVAL; + + virMacAddrGetRaw(macaddr, ifla_vf_mac.mac); + + requestError = virNetDevSendVfSetLinkRequest(ifname, IFLA_VF_MAC, + &ifla_vf_mac, sizeof(ifla_vf_mac)); + /* if allowRetry is true and the error was EINVAL, then + * silently return a failure so the caller can retry with a + * different MAC address. */ + if (requestError == -EINVAL && *allowRetry && !virMacAddrCmp(macaddr, &zeroMAC)) { + rc = requestError; + } else if (requestError) { + /* other errors are permanent */ + virReportSystemError(-requestError, + _("Cannot set interface MAC to %s " + "for ifname %s vf %d"), + (macaddr + ? virMacAddrFormat(macaddr, macstr) + : "(unchanged)"), + ifname ? ifname : "(unspecified)", + vf); + *allowRetry = false; /* no use retrying */ + rc = requestError; + } else { + rc = 0; + } + VIR_DEBUG("RTM_SETLINK %s vf %d MAC=%s - %s", + ifname, vf, + macaddr ? virMacAddrFormat(macaddr, macstr) : "(unchanged)", + rc < 0 ? "Fail" : "Success"); + return rc; +} + +int +virNetDevSetVfConfig(const char *ifname, int vf, + const virMacAddr *macaddr, int vlanid, + bool *allowRetry) +{ + int rc = 0; + if ((rc = virNetDevSetVfMac(ifname, vf, macaddr, allowRetry)) < 0) + return rc; + if ((rc = virNetDevSetVfVlan(ifname, vf, vlanid)) < 0) + return rc; + return rc; +} + /** * virNetDevParseVfInfo: * Get the VF interface information from kernel by netlink, To make netlink @@ -2409,6 +2453,44 @@ virNetDevVFInterfaceStats(virPCIDeviceAddress *vfAddr G_GNUC_UNUSED, return -1; } +int +virNetDevSendVfSetLinkRequest(const char *ifname G_GNUC_UNUSED, int vfInfoType G_GNUC_UNUSED, + const void *payload G_GNUC_UNUSED, + const size_t payloadLen G_GNUC_UNUSED) +{ + virReportSystemError(ENOSYS, "%s", + _("Unable to send a VF SETLINK request on this platform")); + return -1; +} + +int +virNetDevSetVfVlan(const char *ifname G_GNUC_UNUSED, int vf G_GNUC_UNUSED, int vlanid G_GNUC_UNUSED) +{ + virReportSystemError(ENOSYS, "%s", + _("Unable to set a VF VLAN on this platform")); + return -1; +} + +int +virNetDevSetVfMac(const char *ifname G_GNUC_UNUSED, int vf G_GNUC_UNUSED, + const virMacAddr *macaddr G_GNUC_UNUSED, + bool *allowRetry G_GNUC_UNUSED) +{ + virReportSystemError(ENOSYS, "%s", + _("Unable to set a VF MAC on this platform")); + return -1; +} + +int +virNetDevSetVfConfig(const char *ifname G_GNUC_UNUSED, int vf G_GNUC_UNUSED, + const virMacAddr *macaddr G_GNUC_UNUSED, int vlanid G_GNUC_UNUSED, + bool *allowRetry G_GNUC_UNUSED) +{ + virReportSystemError(ENOSYS, "%s", + _("Unable to set a VF config on this platform")); + return -1; +} + #endif /* defined(WITH_LIBNL) */ diff --git a/src/util/virnetdevpriv.h b/src/util/virnetdevpriv.h new file mode 100644 index 0000000000..7990bf5269 --- /dev/null +++ b/src/util/virnetdevpriv.h @@ -0,0 +1,44 @@ +/* + * virnetdevpriv.h: private virnetdev header for unit testing + * + * Copyright (C) 2021 Canonical Ltd. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; If not, see + * <http://www.gnu.org/licenses/>. + */ + +#ifndef LIBVIRT_VIRNETDEVPRIV_H_ALLOW +# error "virnetdevpriv.h may only be included by virnetdev.c or test suites" +#endif /* LIBVIRT_VIRNETDEVPRIV_H_ALLOW */ + +#pragma once + +#include "virnetdev.h" + +int +virNetDevSendVfSetLinkRequest(const char *ifname, int vfInfoType, + const void *payload, const size_t payloadLen); + +int +virNetDevSetVfVlan(const char *ifname, int vf, int vlanid); + +int +virNetDevSetVfMac(const char *ifname, int vf, + const virMacAddr *macaddr, + bool *allowRetry); + +int +virNetDevSetVfConfig(const char *ifname, int vf, + const virMacAddr *macaddr, int vlanid, + bool *allowRetry); diff --git a/tests/virnetdevtest.c b/tests/virnetdevtest.c index aadbeb1ef4..c8dba05c31 100644 --- a/tests/virnetdevtest.c +++ b/tests/virnetdevtest.c @@ -18,11 +18,17 @@ #include <config.h> +#include "internal.h" #include "testutils.h" +#include "virnetlink.h" + +#define LIBVIRT_VIRNETDEVPRIV_H_ALLOW + #ifdef __linux__ -# include "virnetdev.h" +# include "virmock.h" +# include "virnetdevpriv.h" # define VIR_FROM_THIS VIR_FROM_NONE @@ -59,6 +65,215 @@ testVirNetDevGetLinkInfo(const void *opaque) return 0; } +# if defined(WITH_LIBNL) + +int +(*real_virNetDevSendVfSetLinkRequest)(const char *ifname, int vfInfoType, + const void *payload, const size_t payloadLen); + +int +(*real_virNetDevSetVfMac)(const char *ifname, int vf, const virMacAddr *macaddr, bool *allowRetry); + +int +(*real_virNetDevSetVfVlan)(const char *ifname, int vf, int vlanid); + +static void +init_syms(void) +{ + VIR_MOCK_REAL_INIT(virNetDevSendVfSetLinkRequest); + VIR_MOCK_REAL_INIT(virNetDevSetVfMac); + VIR_MOCK_REAL_INIT(virNetDevSetVfVlan); +} + +int +virNetDevSetVfMac(const char *ifname, int vf, + const virMacAddr *macaddr, + bool *allowRetry) +{ + init_syms(); + + if (STREQ_NULLABLE(ifname, "fakeiface-macerror")) { + return -EBUSY; + } else if (STREQ_NULLABLE(ifname, "fakeiface-altmacerror")) { + return -EINVAL; + } else if (STREQ_NULLABLE(ifname, "fakeiface-macerror-novlanerror")) { + return -EAGAIN; + } else if (STREQ_NULLABLE(ifname, "fakeiface-macerror-vlanerror")) { + return -ENODEV; + } else if (STREQ_NULLABLE(ifname, "fakeiface-nomacerror-vlanerror")) { + return 0; + } else if (STREQ_NULLABLE(ifname, "fakeiface-nomacerror-novlanerror")) { + return 0; + } + return real_virNetDevSetVfMac(ifname, vf, macaddr, allowRetry); +} + +int +virNetDevSetVfVlan(const char *ifname, int vf, int vlanid) +{ + init_syms(); + + if (STREQ_NULLABLE(ifname, "fakeiface-macerror-vlanerror")) { + return -EPERM; + } else if (STREQ_NULLABLE(ifname, "fakeiface-nomacerror-vlanerror")) { + return -EPERM; + } else if (STREQ_NULLABLE(ifname, "fakeiface-macerror-novlanerror")) { + return 0; + } else if (STREQ_NULLABLE(ifname, "fakeiface-nomacerror-novlanerror")) { + return 0; + } + return real_virNetDevSetVfVlan(ifname, vf, vlanid); +} + +int +virNetDevSendVfSetLinkRequest(const char *ifname, int vfInfoType, + const void *payload, const size_t payloadLen) +{ + init_syms(); + + if (STREQ_NULLABLE(ifname, "fakeiface-eperm")) { + return -EPERM; + } else if (STREQ_NULLABLE(ifname, "fakeiface-eagain")) { + return -EAGAIN; + } else if (STREQ_NULLABLE(ifname, "fakeiface-einval")) { + return -EINVAL; + } else if (STREQ_NULLABLE(ifname, "fakeiface-ok")) { + return 0; + } + return real_virNetDevSendVfSetLinkRequest(ifname, vfInfoType, payload, payloadLen); +} + +static int +testVirNetDevSetVfMac(const void *opaque G_GNUC_UNUSED) +{ + struct testCase { + const char *ifname; + const int vf_num; + const virMacAddr macaddr; + bool allow_retry; + const int rc; + }; + size_t i = 0; + int rc = 0; + struct testCase testCases[] = { + { .ifname = "fakeiface-ok", .vf_num = 1, + .macaddr = { .addr = { 0, 0, 0, 0, 0, 0 } }, .allow_retry = false, .rc = 0 }, + { .ifname = "fakeiface-ok", .vf_num = 2, + .macaddr = { .addr = { 0, 0, 0, 7, 7, 7 } }, .allow_retry = false, .rc = 0 }, + { .ifname = "fakeiface-ok", .vf_num = 3, + .macaddr = { .addr = { 0, 0, 0, 0, 0, 0 } }, .allow_retry = true, .rc = 0 }, + { .ifname = "fakeiface-ok", .vf_num = 4, + .macaddr = { .addr = { 0, 0, 0, 7, 7, 7 } }, .allow_retry = true, .rc = 0 }, + { .ifname = "fakeiface-eperm", .vf_num = 5, + .macaddr = { .addr = { 0, 0, 0, 0, 0, 0 } }, .allow_retry = false, .rc = -EPERM }, + { .ifname = "fakeiface-einval", .vf_num = 6, + .macaddr = { .addr = { 0, 0, 0, 0, 0, 0 } }, .allow_retry = false, .rc = -EINVAL }, + { .ifname = "fakeiface-einval", .vf_num = 7, + .macaddr = { .addr = { 0, 0, 0, 0, 0, 0 } }, .allow_retry = true, .rc = -EINVAL }, + { .ifname = "fakeiface-einval", .vf_num = 8, + .macaddr = { .addr = { 0, 0, 0, 7, 7, 7 } }, .allow_retry = false, .rc = -EINVAL }, + { .ifname = "fakeiface-einval", .vf_num = 9, + .macaddr = { .addr = { 0, 0, 0, 7, 7, 7 } }, .allow_retry = true, .rc = -EINVAL }, + }; + + for (i = 0; i < sizeof(testCases) / sizeof(struct testCase); ++i) { + rc = virNetDevSetVfMac(testCases[i].ifname, testCases[i].vf_num, + &testCases[i].macaddr, &testCases[i].allow_retry); + if (rc != testCases[i].rc) { + return -1; + } + } + return 0; +} + +static int +testVirNetDevSetVfMissingMac(const void *opaque G_GNUC_UNUSED) +{ + bool allowRetry = false; + /* NULL MAC pointer. */ + if (virNetDevSetVfMac("fakeiface-ok", 1, NULL, &allowRetry) != -EINVAL) { + return -1; + } + allowRetry = true; + if (virNetDevSetVfMac("fakeiface-ok", 1, NULL, &allowRetry) != -EINVAL) { + return -1; + } + return 0; +} + +static int +testVirNetDevSetVfVlan(const void *opaque G_GNUC_UNUSED) +{ + struct testCase { + const char *ifname; + const int vf_num; + const int vlan_id; + const int rc; + }; + size_t i = 0; + int rc = 0; + const struct testCase testCases[] = { + /* VLAN ID is out of range of valid values (0-4095). */ + { .ifname = "enxdeadbeefcafe", .vf_num = 1, .vlan_id = 4096, .rc = -ERANGE }, + { .ifname = "enxdeadbeefcafe", .vf_num = 1, .vlan_id = -1, .rc = -ERANGE }, + { .ifname = "fakeiface-eperm", .vf_num = 1, .vlan_id = 0, .rc = -EPERM }, + { .ifname = "fakeiface-eagain", .vf_num = 1, .vlan_id = 0, .rc = -EAGAIN }, + /* Successful requests with vlan id 0 need to have a zero return code. */ + { .ifname = "fakeiface-ok", .vf_num = 1, .vlan_id = 0, .rc = 0 }, + /* Requests with a non-zero VLAN ID that result in an EPERM need to result in failures. + * failures. */ + { .ifname = "fakeiface-eperm", .vf_num = 1, .vlan_id = 42, .rc = -EPERM }, + /* Requests with a non-zero VLAN ID that result in some other errors need to result in + * failures. */ + { .ifname = "fakeiface-eagain", .vf_num = 1, .vlan_id = 42, .rc = -EAGAIN }, + /* Successful requests with a non-zero VLAN ID */ + { .ifname = "fakeiface-ok", .vf_num = 1, .vlan_id = 42, .rc = 0 }, + }; + + for (i = 0; i < sizeof(testCases) / sizeof(struct testCase); ++i) { + rc = virNetDevSetVfVlan(testCases[i].ifname, testCases[i].vf_num, testCases[i].vlan_id); + if (rc != testCases[i].rc) { + return -1; + } + } + + return 0; +} + +static int +testVirNetDevSetVfConfig(const void *opaque G_GNUC_UNUSED) +{ + struct testCase { + const char *ifname; + const int rc; + }; + int rc = 0; + size_t i = 0; + /* Nested functions are mocked so dummy values are used. */ + const virMacAddr mac = { .addr = { 0xDE, 0xAD, 0xBE, 0xEF, 0xCA, 0xFE }}; + const int vfNum = 1; + const int vlanid = 0; + bool *allowRetry = NULL; + + const struct testCase testCases[] = { + { .ifname = "fakeiface-macerror", .rc = -EBUSY }, + { .ifname = "fakeiface-altmacerror", .rc = -EINVAL }, + { .ifname = "fakeiface-macerror-novlanerror", .rc = -EAGAIN }, + { .ifname = "fakeiface-macerror-vlanerror", .rc = -ENODEV }, + { .ifname = "fakeiface-nomacerror-novlanerror", .rc = 0 }, + }; + + for (i = 0; i < sizeof(testCases) / sizeof(struct testCase); ++i) { + rc = virNetDevSetVfConfig(testCases[i].ifname, vfNum, &mac, vlanid, allowRetry); + if (rc != testCases[i].rc) { + return -1; + } + } + return 0; +} + +# endif /* defined(WITH_LIBNL) */ + static int mymain(void) { @@ -76,6 +291,19 @@ mymain(void) DO_TEST_LINK("lo", VIR_NETDEV_IF_STATE_UNKNOWN, 0); DO_TEST_LINK("eth0-broken", VIR_NETDEV_IF_STATE_DOWN, 0); +# if defined(WITH_LIBNL) + + if (virTestRun("Set VF MAC", testVirNetDevSetVfMac, NULL) < 0) + ret = -1; + if (virTestRun("Set VF MAC: missing MAC pointer", testVirNetDevSetVfMissingMac, NULL) < 0) + ret = -1; + if (virTestRun("Set VF VLAN", testVirNetDevSetVfVlan, NULL) < 0) + ret = -1; + if (virTestRun("Set VF Config", testVirNetDevSetVfConfig, NULL) < 0) + ret = -1; + +# endif /* defined(WITH_LIBNL) */ + return ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE; } -- 2.32.0
There should be a way to show no intent in programming a VLAN at all (including clearing it). This allows handling error conditions differently when VLAN clearing is explicit (vlan id == 0) vs implicit (vlanid == NULL - try to clear it if possible). Signed-off-by: Dmitrii Shcherbakov <dmitrii.shcherbakov@canonical.com> --- src/hypervisor/virhostdev.c | 4 +++- src/util/virnetdev.c | 41 ++++++++++++++++++++++++------------- src/util/virnetdevpriv.h | 4 ++-- tests/virnetdevtest.c | 27 ++++++++++++++++++++---- 4 files changed, 55 insertions(+), 21 deletions(-) diff --git a/src/hypervisor/virhostdev.c b/src/hypervisor/virhostdev.c index e44eb7f1d3..929b0a0805 100644 --- a/src/hypervisor/virhostdev.c +++ b/src/hypervisor/virhostdev.c @@ -463,6 +463,7 @@ virHostdevSetNetConfig(virDomainHostdevDef *hostdev, const virNetDevVPortProfile *virtPort; int vf = -1; bool port_profile_associate = true; + bool setVlan = false; if (!virHostdevIsPCINetDevice(hostdev)) return 0; @@ -471,6 +472,7 @@ virHostdevSetNetConfig(virDomainHostdevDef *hostdev, return -1; vlan = virDomainNetGetActualVlan(hostdev->parentnet); + setVlan = vlan != NULL; virtPort = virDomainNetGetActualVirtPortProfile(hostdev->parentnet); if (virtPort) { if (vlan) { @@ -486,7 +488,7 @@ virHostdevSetNetConfig(virDomainHostdevDef *hostdev, return -1; } else { if (virNetDevSetNetConfig(linkdev, vf, &hostdev->parentnet->mac, - vlan, NULL, true) < 0) + vlan, NULL, setVlan) < 0) return -1; } diff --git a/src/util/virnetdev.c b/src/util/virnetdev.c index ea8cae0dcf..d4d505f7c1 100644 --- a/src/util/virnetdev.c +++ b/src/util/virnetdev.c @@ -1607,20 +1607,25 @@ virNetDevSendVfSetLinkRequest(const char *ifname, int vfInfoType, } int -virNetDevSetVfVlan(const char *ifname, int vf, int vlanid) +virNetDevSetVfVlan(const char *ifname, int vf, const int *vlanid) { int rc = 0; int requestError = 0; struct ifla_vf_vlan ifla_vf_vlan = { .vf = vf, - .vlan = vlanid, .qos = 0, + /* If vlanid is NULL, assume it needs to be cleared. */ + .vlan = 0, }; - /* VLAN ids 0 and 4095 are reserved per 802.1Q but are valid values. */ - if ((vlanid < 0 || vlanid > 4095)) { - return -ERANGE; + if (vlanid != NULL) { + /* VLAN ids 0 and 4095 are reserved per 802.1Q but are valid values. */ + if ((*vlanid < 0 || *vlanid > 4095)) { + return -ERANGE; + } else { + ifla_vf_vlan.vlan = *vlanid; + } } requestError = virNetDevSendVfSetLinkRequest(ifname, IFLA_VF_VLAN, @@ -1630,12 +1635,12 @@ virNetDevSetVfVlan(const char *ifname, int vf, int vlanid) virReportSystemError(-requestError, _("Cannot set interface vlanid to %d " "for ifname %s vf %d"), - vlanid, ifname ? ifname : "(unspecified)", vf); + ifla_vf_vlan.vlan, ifname ? ifname : "(unspecified)", vf); rc = requestError; } VIR_DEBUG("RTM_SETLINK %s vf %d vlanid=%d - %s", ifname, vf, - vlanid, rc < 0 ? "Fail" : "Success"); + ifla_vf_vlan.vlan, rc < 0 ? "Fail" : "Success"); return rc; } @@ -1689,7 +1694,7 @@ virNetDevSetVfMac(const char *ifname, int vf, int virNetDevSetVfConfig(const char *ifname, int vf, - const virMacAddr *macaddr, int vlanid, + const virMacAddr *macaddr, const int *vlanid, bool *allowRetry) { int rc = 0; @@ -2225,7 +2230,7 @@ virNetDevSetNetConfig(const char *linkdev, int vf, const char *pfDevName = NULL; g_autofree char *pfDevOrig = NULL; g_autofree char *vfDevOrig = NULL; - int vlanTag = -1; + g_autofree int *vlanTag = NULL; g_autoptr(virPCIDevice) vfPCIDevice = NULL; if (vf >= 0) { @@ -2284,10 +2289,17 @@ virNetDevSetNetConfig(const char *linkdev, int vf, return -1; } - vlanTag = vlan->tag[0]; + vlanTag = g_new0(int, 1); + *vlanTag = vlan->tag[0]; } else if (setVlan) { - vlanTag = 0; /* assure any existing vlan tag is reset */ + vlanTag = g_new0(int, 1); + /* Assure any existing vlan tag is reset. */ + *vlanTag = 0; + } else { + /* Indicate that setting a VLAN has not been explicitly requested. + * This allows selected errors in clearing a VF VLAN to be ignored. */ + vlanTag = NULL; } } @@ -2369,7 +2381,7 @@ virNetDevSetNetConfig(const char *linkdev, int vf, } } - if (adminMAC || vlanTag >= 0) { + if (adminMAC) { /* Set vlanTag and admin MAC using an RTM_SETLINK request sent to * PFdevname+VF#, if mac != NULL this will set the "admin MAC" via * the PF, *not* the actual VF MAC - the admin MAC only takes @@ -2464,7 +2476,8 @@ virNetDevSendVfSetLinkRequest(const char *ifname G_GNUC_UNUSED, int vfInfoType G } int -virNetDevSetVfVlan(const char *ifname G_GNUC_UNUSED, int vf G_GNUC_UNUSED, int vlanid G_GNUC_UNUSED) +virNetDevSetVfVlan(const char *ifname G_GNUC_UNUSED, int vf G_GNUC_UNUSED, + const int *vlanid G_GNUC_UNUSED) { virReportSystemError(ENOSYS, "%s", _("Unable to set a VF VLAN on this platform")); @@ -2483,7 +2496,7 @@ virNetDevSetVfMac(const char *ifname G_GNUC_UNUSED, int vf G_GNUC_UNUSED, int virNetDevSetVfConfig(const char *ifname G_GNUC_UNUSED, int vf G_GNUC_UNUSED, - const virMacAddr *macaddr G_GNUC_UNUSED, int vlanid G_GNUC_UNUSED, + const virMacAddr *macaddr G_GNUC_UNUSED, const int *vlanid G_GNUC_UNUSED, bool *allowRetry G_GNUC_UNUSED) { virReportSystemError(ENOSYS, "%s", diff --git a/src/util/virnetdevpriv.h b/src/util/virnetdevpriv.h index 7990bf5269..84a42fb747 100644 --- a/src/util/virnetdevpriv.h +++ b/src/util/virnetdevpriv.h @@ -31,7 +31,7 @@ virNetDevSendVfSetLinkRequest(const char *ifname, int vfInfoType, const void *payload, const size_t payloadLen); int -virNetDevSetVfVlan(const char *ifname, int vf, int vlanid); +virNetDevSetVfVlan(const char *ifname, int vf, const int *vlanid); int virNetDevSetVfMac(const char *ifname, int vf, @@ -40,5 +40,5 @@ virNetDevSetVfMac(const char *ifname, int vf, int virNetDevSetVfConfig(const char *ifname, int vf, - const virMacAddr *macaddr, int vlanid, + const virMacAddr *macaddr, const int *vlanid, bool *allowRetry); diff --git a/tests/virnetdevtest.c b/tests/virnetdevtest.c index c8dba05c31..f5f54653bb 100644 --- a/tests/virnetdevtest.c +++ b/tests/virnetdevtest.c @@ -75,7 +75,7 @@ int (*real_virNetDevSetVfMac)(const char *ifname, int vf, const virMacAddr *macaddr, bool *allowRetry); int -(*real_virNetDevSetVfVlan)(const char *ifname, int vf, int vlanid); +(*real_virNetDevSetVfVlan)(const char *ifname, int vf, const int *vlanid); static void init_syms(void) @@ -109,7 +109,7 @@ virNetDevSetVfMac(const char *ifname, int vf, } int -virNetDevSetVfVlan(const char *ifname, int vf, int vlanid) +virNetDevSetVfVlan(const char *ifname, int vf, const int *vlanid) { init_syms(); @@ -210,6 +210,11 @@ testVirNetDevSetVfVlan(const void *opaque G_GNUC_UNUSED) const int vlan_id; const int rc; }; + struct nullVlanTestCase { + const char *ifname; + const int vf_num; + const int rc; + }; size_t i = 0; int rc = 0; const struct testCase testCases[] = { @@ -230,13 +235,27 @@ testVirNetDevSetVfVlan(const void *opaque G_GNUC_UNUSED) { .ifname = "fakeiface-ok", .vf_num = 1, .vlan_id = 42, .rc = 0 }, }; + const struct nullVlanTestCase nullVLANTestCases[] = { + { .ifname = "fakeiface-eperm", .vf_num = 1, .rc = -EPERM }, + { .ifname = "fakeiface-eagain", .vf_num = 1, .rc = -EAGAIN }, + /* Successful requests with vlan id 0 need to have a zero return code. */ + { .ifname = "fakeiface-ok", .vf_num = 1, .rc = 0 }, + }; + for (i = 0; i < sizeof(testCases) / sizeof(struct testCase); ++i) { - rc = virNetDevSetVfVlan(testCases[i].ifname, testCases[i].vf_num, testCases[i].vlan_id); + rc = virNetDevSetVfVlan(testCases[i].ifname, testCases[i].vf_num, &testCases[i].vlan_id); if (rc != testCases[i].rc) { return -1; } } + for (i = 0; i < sizeof(nullVLANTestCases) / sizeof(struct nullVlanTestCase); ++i) { + rc = virNetDevSetVfVlan(nullVLANTestCases[i].ifname, nullVLANTestCases[i].vf_num, NULL); + if (rc != nullVLANTestCases[i].rc) { + return -1; + } + } + return 0; } @@ -264,7 +283,7 @@ testVirNetDevSetVfConfig(const void *opaque G_GNUC_UNUSED) }; for (i = 0; i < sizeof(testCases) / sizeof(struct testCase); ++i) { - rc = virNetDevSetVfConfig(testCases[i].ifname, vfNum, &mac, vlanid, allowRetry); + rc = virNetDevSetVfConfig(testCases[i].ifname, vfNum, &mac, &vlanid, allowRetry); if (rc != testCases[i].rc) { return -1; } -- 2.32.0
SmartNIC DPUs may not expose some privileged eswitch operations to the hypervisor hosts. For example, this happens with Bluefield devices running in the ECPF (default) mode for security reasons. While VF MAC address programming is possible via an RTM_SETLINK operation, trying to set a VLAN ID in the same operation will fail with EPERM. The equivalent ip link commands below provide an illustration: 1. This works: sudo ip link set enp130s0f0 vf 2 mac de:ad:be:ef:ca:fe 2. Setting (or clearing) a VLAN fails with EPERM: sudo ip link set enp130s0f0 vf 2 vlan 0 RTNETLINK answers: Operation not permitted 3. This is what Libvirt attempts to do today (when trying to clear a VF VLAN at the same time as programming a VF MAC). sudo ip link set enp130s0f0 vf 2 vlan 0 mac de:ad:be:ef:ca:fe RTNETLINK answers: Operation not permitted If setting an explicit VLAN ID results in an EPERM, clearing a VLAN (setting a VLAN ID to 0) can be handled gracefully by ignoring the EPERM error with the rationale being that if we cannot set this state in the first place, we cannot clear it either. In order to keep explicit clearing of VLAN ID working as it used to be passing a NULL pointer for VLAN ID is used. Signed-off-by: Dmitrii Shcherbakov <dmitrii.shcherbakov@canonical.com> --- NEWS.rst | 14 ++++++++++++++ src/util/virnetdev.c | 11 ++++++++++- tests/virnetdevtest.c | 2 +- 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/NEWS.rst b/NEWS.rst index 4d690ff64b..89d7e54f32 100644 --- a/NEWS.rst +++ b/NEWS.rst @@ -19,6 +19,20 @@ v8.0.0 (unreleased) * **Improvements** + * virnetdev: Ignore EPERM on implicit clearing of VF VLAN ID + + Libvirt will now ignore EPERM errors on attempts to implicitly clear a + VLAN ID (when a VLAN is not explicitly provided via an interface XML + using a 0 or a non-zero value) as SmartNIC DPUs do not expose VLAN + programming capabilities to the hypervisor host. This allows Libvirt + clients to avoid specifying a VLAN and expect VF configuration to work + since Libvirt tries to clear a VLAN in the same operation + as setting a MAC address for VIR_DOMAIN_NET_TYPE_HOSTDEV devices which + is now split into two distinct operations. EPERM errors received while + trying to program a non-zero VLAN ID or explicitly program a VLAN ID 0 + will still cause errors as before so there is no change in behavior + in those cases. + * **Bug fixes** diff --git a/src/util/virnetdev.c b/src/util/virnetdev.c index d4d505f7c1..8031ebb17f 100644 --- a/src/util/virnetdev.c +++ b/src/util/virnetdev.c @@ -1631,12 +1631,21 @@ virNetDevSetVfVlan(const char *ifname, int vf, const int *vlanid) requestError = virNetDevSendVfSetLinkRequest(ifname, IFLA_VF_VLAN, &ifla_vf_vlan, sizeof(ifla_vf_vlan)); - if (requestError) { + /* If vlanid is NULL - we are attempting to implicitly clear an existing VLAN id. + * An EPERM received at this stage is an indicator that the embedded + * switch is not exposed to this host and the network driver is not + * able to set a VLAN for a VF, whereas the Libvirt client has not + * explicitly configured a VLAN or requested it to be cleared via vid 0. */ + if (requestError == -EPERM && vlanid == NULL) { + rc = 0; + } else if (requestError) { virReportSystemError(-requestError, _("Cannot set interface vlanid to %d " "for ifname %s vf %d"), ifla_vf_vlan.vlan, ifname ? ifname : "(unspecified)", vf); rc = requestError; + } else { + rc = 0; } VIR_DEBUG("RTM_SETLINK %s vf %d vlanid=%d - %s", ifname, vf, diff --git a/tests/virnetdevtest.c b/tests/virnetdevtest.c index f5f54653bb..d73eaec817 100644 --- a/tests/virnetdevtest.c +++ b/tests/virnetdevtest.c @@ -236,7 +236,7 @@ testVirNetDevSetVfVlan(const void *opaque G_GNUC_UNUSED) }; const struct nullVlanTestCase nullVLANTestCases[] = { - { .ifname = "fakeiface-eperm", .vf_num = 1, .rc = -EPERM }, + { .ifname = "fakeiface-eperm", .vf_num = 1, .rc = 0 }, { .ifname = "fakeiface-eagain", .vf_num = 1, .rc = -EAGAIN }, /* Successful requests with vlan id 0 need to have a zero return code. */ { .ifname = "fakeiface-ok", .vf_num = 1, .rc = 0 }, -- 2.32.0
participants (1)
-
Dmitrii Shcherbakov