Re: Storing PCI(e) VPD Board Serial Numbers
On Fri, May 28, 2021 at 10:58:05PM +0300, Dmitrii Shcherbakov wrote:
Hello Libvirt Developers,
I am looking for some feedback on a planned enhancement to Libvirt: the aim
is
to store a portion of PCI(e) Vital Product Data (VPD) for each device along
with other PCI/PCIe device information already collected. Specifically, the
SN
(Serial Number) read-only field of a VPD data structure of a device is of
interest which is described in PCI/PCIe specs (PCI local bus 2.1+ and PCIe
4.0+).
The context for this is the cross-project work in OpenStack (Nova, Neutron),
OVS and OVN to support for off-path SmartNIC DPUs ([1], [2], [3], [4]). The
Nova specification [1] provides an overview of the relevant hardware and the
use-case for board serial numbers, however, VPD is the standard capability
in
the PCI/PCIe specifications not tied to the use-case in particular so the
suggestion from the Nova core team was to aim at introducing means of
collecting this information via Libvirt. It can then be retrieved by the
respective virt driver in Nova via Libvirt without having to introduce this
code into Nova itself.
I've talked with Sean Mooney at little about the use case this morning.
IIUC, the high level scenario is as follows
- The main host machine has a PCI controller topology to which the
NICs are attached. This is how the host OS and by extension libvirt,
nova, etc, see the PCI devices.
- There is a second PCI controller topology to which the NICs are
attached. This is only visible to the arm cores for the offload
engine
- Nova/Neutron can identify the NICs based on the PCI topology
seen by the host OS, but need to tell the NIC mgmt software
which NIC to use in a way that can be undersood by the offload
cores.
IOW, the PCI address is not usable as a unique identifier because
there are two completely independant PCI topologies with no mapping
between them.
The VPD data provides a replacement way to identify a NIC based on
a unique serial number that is indendant of PCI topology. Nova needs
this serial number in order to configure the device offload featues.
This seems like a reasonable feature request to me, since there is
a piece of info that apps using libvirt need, and libvirt does not
expose this. Requiring the mgmt app like Nova to dig into the host
PCI config space indicates a clear gap in libvirt functionality.
I would like to suggest the following to be done in Libvirt:
1) adding the code for extracting a serial number from VPD for PCI/PCIe
devices
in general and storing it for exposure via the Libvirt API;
More specifically, I propose adding a nested capability called "vpd" under
VIR_NODE_DEV_CAP_PCI_DEV:
<capability type='pci'>
<capability type='vpd'>
<serial>UNIQUESERIAL</serial>
<!-- ... other VPD attributes if present -->
</capability>
<!-- ... -->
</capability>
This looks like a reasonable proposal
2) (optional) implementing functionality to obtain a board serial
number via
devlink-info for PCIe devices if they do not expose a VPD capability
but the device driver can retrieve it via firmware. The board serial number
can be stored in the same element as suggested above.
Is scenario (2) going to be at all common ? What would be a reason why the
info is not exposed via the standardized VPD - is it just a legacy hardware
issue ?
Not all devices expose the devlink API and even fewer do expose board
serial
via devlink-info:
* devlink was added in 4.10 [11];
* devlink-info was introduced in 5.1 [12];
* querying for board.serial_number was added in kernel 5.9 [13] and iproute2
5.9.0 [14];
* Besides the generic devlink infrastructure support above, device drivers
also need to support exposing this field.
Therefore, implementing two approaches (sysfs VPD, devlink) is preferable
for better compatibility.
I would appreciate any feedback on whether this potential addition makes
sense.
If so, I can look into implementing this.
It makes sense to me.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|