[PATCH v2 0/3] Add virt machine support for configuring PCI high memory MMIO size
This patch series adds support for configuring the PCI high memory MMIO window size for aarch64 virt machine types using the highmem-mmio-size feature introduced in QEMU v10.0.0 [1]. It allows users to configure the size of the high memory MMIO window above 4GB, which can be required to support PCI passthrough with devices that have a large BARs. The feature is exposed through the existing pcihole64 element associated with the PCIe root controller: <controller type='pci' index='0' model='pcie-root'> <pcihole64 unit='GiB'>512</pcihole64> </controller> This existing schema supports the same semantics for the QEMU PC machine pci-hole64-size parameter and is a natural fit for supporting the highmem-mmio-size feature on the aarch64 virt machine. This series is applied over master and depends on the recently merged patch [2] that added support for QEMU v10.0.0 aarch64 capabilities. For your convenience, this series is also available on Github [3]. [1] https://github.com/qemu/qemu/commit/f10104aeae3a17f181d5bb37b7fd7dad7fe86cba [2] https://github.com/nvmochs/libvirt/commit/cea2ee1d28780808172911e5c586478d4e... [3] git fetch https://github.com/nvmochs/libvirt.git pci_highmem_mmio_size_pcihole64 Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> Changelog: v2 - Use existing XML scmhema (pcihole64 element) instead of a new one Matthew R. Ochs (3): qemu: Add capability for PCI high memory MMIO size qemu: Add command line support for PCI high memory MMIO size tests: Add pcihole64 test for virt machine src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_command.c | 14 ++++++++- src/qemu/qemu_validate.c | 8 +++-- .../caps_10.0.0_aarch64.xml | 1 + .../pcihole64-virt.aarch64-latest.args | 31 +++++++++++++++++++ .../pcihole64-virt.aarch64-latest.xml | 29 +++++++++++++++++ tests/qemuxmlconfdata/pcihole64-virt.xml | 17 ++++++++++ tests/qemuxmlconftest.c | 1 + 9 files changed, 101 insertions(+), 3 deletions(-) create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.xml -- 2.46.0
Add QEMU capability for PCI high memory MMIO size configuration: - Add QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE capability - Add capability to virt machine properties - Add highmem-mmio-size virt machine property to aarch64 qemu 10.0.0 capabilities This allows detecting support for the highmem-mmio-size virt machine property in QEMU. Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> --- src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + tests/qemucapabilitiesdata/caps_10.0.0_aarch64.xml | 1 + 3 files changed, 4 insertions(+) diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c index c24584bf75fd..b600dd38b8fc 100644 --- a/src/qemu/qemu_capabilities.c +++ b/src/qemu/qemu_capabilities.c @@ -732,6 +732,7 @@ VIR_ENUM_IMPL(virQEMUCaps, /* 475 */ "virtio-scsi.iothread-mapping", /* QEMU_CAPS_VIRTIO_SCSI_IOTHREAD_MAPPING */ + "machine.virt.highmem-mmio-size", /* QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE */ ); @@ -1771,6 +1772,7 @@ static struct virQEMUCapsStringFlags virQEMUCapsMachinePropsVirt[] = { { "iommu", QEMU_CAPS_MACHINE_VIRT_IOMMU }, { "ras", QEMU_CAPS_MACHINE_VIRT_RAS }, { "aia", QEMU_CAPS_MACHINE_VIRT_AIA }, + { "highmem-mmio-size", QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE }, }; static struct virQEMUCapsStringFlags virQEMUCapsMachinePropsGeneric[] = { diff --git a/src/qemu/qemu_capabilities.h b/src/qemu/qemu_capabilities.h index 40aa7acef42e..df33212a348a 100644 --- a/src/qemu/qemu_capabilities.h +++ b/src/qemu/qemu_capabilities.h @@ -713,6 +713,7 @@ typedef enum { /* virQEMUCapsFlags grouping marker for syntax-check */ /* 475 */ QEMU_CAPS_VIRTIO_SCSI_IOTHREAD_MAPPING, /* virtio-scsi supports per-virtqueue iothread mapping */ + QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE, /* -machine virt,highmem-mmio-size=<size> */ QEMU_CAPS_LAST /* this must always be the last item */ } virQEMUCapsFlags; diff --git a/tests/qemucapabilitiesdata/caps_10.0.0_aarch64.xml b/tests/qemucapabilitiesdata/caps_10.0.0_aarch64.xml index e14e926c290d..7fd50185575a 100644 --- a/tests/qemucapabilitiesdata/caps_10.0.0_aarch64.xml +++ b/tests/qemucapabilitiesdata/caps_10.0.0_aarch64.xml @@ -160,6 +160,7 @@ <flag name='blockdev-set-active'/> <flag name='shim'/> <flag name='virtio-scsi.iothread-mapping'/> + <flag name='machine.virt.highmem-mmio-size'/> <version>10000000</version> <microcodeVersion>61700285</microcodeVersion> <package>v10.0.0</package> -- 2.46.0
On Wed, May 14, 2025 at 05:18:45PM -0700, Matthew R. Ochs wrote:
Add QEMU capability for PCI high memory MMIO size configuration: - Add QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE capability - Add capability to virt machine properties - Add highmem-mmio-size virt machine property to aarch64 qemu 10.0.0 capabilities
This allows detecting support for the highmem-mmio-size virt machine property in QEMU.
Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> --- src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + tests/qemucapabilitiesdata/caps_10.0.0_aarch64.xml | 1 + 3 files changed, 4 insertions(+)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Add support for generating QEMU command line with PCI high memory MMIO size: - Add highmem-mmio-size to machine command line generation using size conveyed through pcihole64 - Add validation for aarch64/virt machine type requirement - Add capability check for QEMU support This enables configuring the PCI high memory MMIO window size for aarch64 virt machine types using the existing pcihole64 element. Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> --- src/qemu/qemu_command.c | 14 +++++++++++++- src/qemu/qemu_validate.c | 8 ++++++-- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index e6d308534f87..1386b2772a55 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -6195,7 +6195,8 @@ qemuBuildGlobalControllerCommandLine(virCommand *cmd, for (i = 0; i < def->ncontrollers; i++) { virDomainControllerDef *cont = def->controllers[i]; if (cont->type == VIR_DOMAIN_CONTROLLER_TYPE_PCI && - cont->opts.pciopts.pcihole64) { + cont->opts.pciopts.pcihole64 && + !qemuDomainIsARMVirt(def)) { const char *hoststr = NULL; switch (cont->model) { @@ -7112,6 +7113,17 @@ qemuBuildMachineCommandLine(virCommand *cmd, qemuBuildMachineACPI(&buf, def, qemuCaps); + if (qemuDomainIsARMVirt(def)) { + for (i = 0; i < def->ncontrollers; i++) { + virDomainControllerDef *cont = def->controllers[i]; + if (cont->type == VIR_DOMAIN_CONTROLLER_TYPE_PCI && + cont->opts.pciopts.pcihole64) { + virBufferAsprintf(&buf, ",highmem-mmio-size=%lluK", cont->opts.pciopts.pcihole64size); + break; + } + } + } + virCommandAddArgBuffer(cmd, &buf); return 0; diff --git a/src/qemu/qemu_validate.c b/src/qemu/qemu_validate.c index b2c3c9e2f631..f808aac1063a 100644 --- a/src/qemu/qemu_validate.c +++ b/src/qemu/qemu_validate.c @@ -4074,7 +4074,9 @@ qemuValidateDomainDeviceDefControllerPCI(const virDomainControllerDef *cont, switch ((virDomainControllerModelPCI) cont->model) { case VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT: if (pciopts->pcihole64 || pciopts->pcihole64size != 0) { - if (!qemuDomainIsI440FX(def)) { + if (!qemuDomainIsI440FX(def) && + !(qemuDomainIsARMVirt(def) && virQEMUCapsGet(qemuCaps, + QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE))) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Setting the 64-bit PCI hole size is not supported for machine '%1$s'"), def->os.machine); @@ -4085,7 +4087,9 @@ qemuValidateDomainDeviceDefControllerPCI(const virDomainControllerDef *cont, case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT: if (pciopts->pcihole64 || pciopts->pcihole64size != 0) { - if (!qemuDomainIsQ35(def)) { + if (!qemuDomainIsQ35(def) && + !(qemuDomainIsARMVirt(def) && virQEMUCapsGet(qemuCaps, + QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE))) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Setting the 64-bit PCI hole size is not supported for machine '%1$s'"), def->os.machine); -- 2.46.0
On Wed, May 14, 2025 at 05:18:50PM -0700, Matthew R. Ochs wrote:
Add support for generating QEMU command line with PCI high memory MMIO size: - Add highmem-mmio-size to machine command line generation using size conveyed through pcihole64 - Add validation for aarch64/virt machine type requirement - Add capability check for QEMU support
This enables configuring the PCI high memory MMIO window size for aarch64 virt machine types using the existing pcihole64 element.
Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> --- src/qemu/qemu_command.c | 14 +++++++++++++- src/qemu/qemu_validate.c | 8 ++++++-- 2 files changed, 19 insertions(+), 3 deletions(-)
diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index e6d308534f87..1386b2772a55 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -6195,7 +6195,8 @@ qemuBuildGlobalControllerCommandLine(virCommand *cmd, for (i = 0; i < def->ncontrollers; i++) { virDomainControllerDef *cont = def->controllers[i]; if (cont->type == VIR_DOMAIN_CONTROLLER_TYPE_PCI && - cont->opts.pciopts.pcihole64) { + cont->opts.pciopts.pcihole64 && + !qemuDomainIsARMVirt(def)) {
This ought to be a positive check ..IsQ35(..) || ..IsI440FX(..)
const char *hoststr = NULL;
switch (cont->model) { @@ -7112,6 +7113,17 @@ qemuBuildMachineCommandLine(virCommand *cmd,
qemuBuildMachineACPI(&buf, def, qemuCaps);
+ if (qemuDomainIsARMVirt(def)) { + for (i = 0; i < def->ncontrollers; i++) { + virDomainControllerDef *cont = def->controllers[i]; + if (cont->type == VIR_DOMAIN_CONTROLLER_TYPE_PCI && + cont->opts.pciopts.pcihole64) { + virBufferAsprintf(&buf, ",highmem-mmio-size=%lluK", cont->opts.pciopts.pcihole64size); + break; + } + } + } + virCommandAddArgBuffer(cmd, &buf);
return 0; diff --git a/src/qemu/qemu_validate.c b/src/qemu/qemu_validate.c index b2c3c9e2f631..f808aac1063a 100644 --- a/src/qemu/qemu_validate.c +++ b/src/qemu/qemu_validate.c @@ -4074,7 +4074,9 @@ qemuValidateDomainDeviceDefControllerPCI(const virDomainControllerDef *cont, switch ((virDomainControllerModelPCI) cont->model) { case VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT: if (pciopts->pcihole64 || pciopts->pcihole64size != 0) { - if (!qemuDomainIsI440FX(def)) { + if (!qemuDomainIsI440FX(def) && + !(qemuDomainIsARMVirt(def) && virQEMUCapsGet(qemuCaps, + QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE))) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Setting the 64-bit PCI hole size is not supported for machine '%1$s'"), def->os.machine);
IIUC arm virt machine is always PCI-E, so we shouldn't allow it for arm here.
@@ -4085,7 +4087,9 @@ qemuValidateDomainDeviceDefControllerPCI(const virDomainControllerDef *cont,
case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_ROOT: if (pciopts->pcihole64 || pciopts->pcihole64size != 0) { - if (!qemuDomainIsQ35(def)) { + if (!qemuDomainIsQ35(def) && + !(qemuDomainIsARMVirt(def) && virQEMUCapsGet(qemuCaps, + QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE))) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Setting the 64-bit PCI hole size is not supported for machine '%1$s'"), def->os.machine); -- 2.46.0
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On May 15, 2025, at 6:54 AM, Daniel P. Berrangé <berrange@redhat.com> wrote: On Wed, May 14, 2025 at 05:18:50PM -0700, Matthew R. Ochs wrote:
Add support for generating QEMU command line with PCI high memory MMIO size: - Add highmem-mmio-size to machine command line generation using size conveyed through pcihole64 - Add validation for aarch64/virt machine type requirement - Add capability check for QEMU support
This enables configuring the PCI high memory MMIO window size for aarch64 virt machine types using the existing pcihole64 element.
Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> --- src/qemu/qemu_command.c | 14 +++++++++++++- src/qemu/qemu_validate.c | 8 ++++++-- 2 files changed, 19 insertions(+), 3 deletions(-)
diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index e6d308534f87..1386b2772a55 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -6195,7 +6195,8 @@ qemuBuildGlobalControllerCommandLine(virCommand *cmd, for (i = 0; i < def->ncontrollers; i++) { virDomainControllerDef *cont = def->controllers[i]; if (cont->type == VIR_DOMAIN_CONTROLLER_TYPE_PCI && - cont->opts.pciopts.pcihole64) { + cont->opts.pciopts.pcihole64 && + !qemuDomainIsARMVirt(def)) {
This ought to be a positive check
..IsQ35(..) || ..IsI440FX(..)
Makes sense, will fix in v3.
diff --git a/src/qemu/qemu_validate.c b/src/qemu/qemu_validate.c index b2c3c9e2f631..f808aac1063a 100644 --- a/src/qemu/qemu_validate.c +++ b/src/qemu/qemu_validate.c @@ -4074,7 +4074,9 @@ qemuValidateDomainDeviceDefControllerPCI(const virDomainControllerDef *cont, switch ((virDomainControllerModelPCI) cont->model) { case VIR_DOMAIN_CONTROLLER_MODEL_PCI_ROOT: if (pciopts->pcihole64 || pciopts->pcihole64size != 0) { - if (!qemuDomainIsI440FX(def)) { + if (!qemuDomainIsI440FX(def) && + !(qemuDomainIsARMVirt(def) && virQEMUCapsGet(qemuCaps, + QEMU_CAPS_MACHINE_VIRT_HIGHMEM_MMIO_SIZE))) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Setting the 64-bit PCI hole size is not supported for machine '%1$s'"), def->os.machine);
IIUC arm virt machine is always PCI-E, so we shouldn't allow it for arm here.
Yep, will remove this change in v3.
Add a pcihole64 test for the aarch64 virt machine that verifies the value is propagated to the highmem-mmio-size virt machine parameter. Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> --- .../pcihole64-virt.aarch64-latest.args | 31 +++++++++++++++++++ .../pcihole64-virt.aarch64-latest.xml | 29 +++++++++++++++++ tests/qemuxmlconfdata/pcihole64-virt.xml | 17 ++++++++++ tests/qemuxmlconftest.c | 1 + 4 files changed, 78 insertions(+) create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.xml diff --git a/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args b/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args new file mode 100644 index 000000000000..d2df31ed868e --- /dev/null +++ b/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args @@ -0,0 +1,31 @@ +LC_ALL=C \ +PATH=/bin \ +HOME=/var/lib/libvirt/qemu/domain--1-foo \ +USER=test \ +LOGNAME=test \ +XDG_DATA_HOME=/var/lib/libvirt/qemu/domain--1-foo/.local/share \ +XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain--1-foo/.cache \ +XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-foo/.config \ +/usr/bin/qemu-system-aarch64 \ +-name guest=foo,debug-threads=on \ +-S \ +-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain--1-foo/master-key.aes"}' \ +-machine virt,usb=off,gic-version=2,dump-guest-core=off,memory-backend=mach-virt.ram,acpi=off,highmem-mmio-size=536870912K \ +-accel tcg \ +-cpu cortex-a15 \ +-m size=1048576k \ +-object '{"qom-type":"memory-backend-ram","id":"mach-virt.ram","size":1073741824}' \ +-overcommit mem-lock=off \ +-smp 1,sockets=1,cores=1,threads=1 \ +-uuid 6ba7b810-9dad-11d1-80b4-00c04fd430c8 \ +-display none \ +-no-user-config \ +-nodefaults \ +-chardev socket,id=charmonitor,fd=1729,server=on,wait=off \ +-mon chardev=charmonitor,id=monitor,mode=control \ +-rtc base=utc \ +-no-shutdown \ +-boot strict=on \ +-audiodev '{"id":"audio1","driver":"none"}' \ +-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ +-msg timestamp=on diff --git a/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml b/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml new file mode 100644 index 000000000000..1fb38b429e9f --- /dev/null +++ b/tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml @@ -0,0 +1,29 @@ +<domain type='qemu'> + <name>foo</name> + <uuid>6ba7b810-9dad-11d1-80b4-00c04fd430c8</uuid> + <memory unit='KiB'>1048576</memory> + <currentMemory unit='KiB'>1048576</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='aarch64' machine='virt'>hvm</type> + <boot dev='hd'/> + </os> + <features> + <gic version='2'/> + </features> + <cpu mode='custom' match='exact' check='none'> + <model fallback='forbid'>cortex-a15</model> + </cpu> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-aarch64</emulator> + <controller type='pci' index='0' model='pcie-root'> + <pcihole64 unit='KiB'>536870912</pcihole64> + </controller> + <audio id='1' type='none'/> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxmlconfdata/pcihole64-virt.xml b/tests/qemuxmlconfdata/pcihole64-virt.xml new file mode 100644 index 000000000000..00bb51ab0e9a --- /dev/null +++ b/tests/qemuxmlconfdata/pcihole64-virt.xml @@ -0,0 +1,17 @@ +<domain type='qemu'> + <name>foo</name> + <uuid>6ba7b810-9dad-11d1-80b4-00c04fd430c8</uuid> + <memory unit='KiB'>1048576</memory> + <currentMemory unit='KiB'>1048576</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='aarch64' machine='virt'>hvm</type> + </os> + <devices> + <emulator>/usr/bin/qemu-system-aarch64</emulator> + <controller type='pci' index='0' model='pcie-root'> + <pcihole64 unit='GiB'>512</pcihole64> + </controller> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxmlconftest.c b/tests/qemuxmlconftest.c index 1f31ec810c7a..7fe51ba4f81c 100644 --- a/tests/qemuxmlconftest.c +++ b/tests/qemuxmlconftest.c @@ -2587,6 +2587,7 @@ mymain(void) DO_TEST_CAPS_LATEST("pcihole64"); DO_TEST_CAPS_LATEST("pcihole64-q35"); + DO_TEST_CAPS_ARCH_LATEST("pcihole64-virt", "aarch64"); DO_TEST_CAPS_ARCH_LATEST("arm-vexpressa9-nodevs", "aarch64"); DO_TEST_CAPS_ARCH_LATEST("arm-vexpressa9-basic", "aarch64"); -- 2.46.0
On Wed, May 14, 2025 at 05:18:54PM -0700, Matthew R. Ochs wrote:
Add a pcihole64 test for the aarch64 virt machine that verifies the value is propagated to the highmem-mmio-size virt machine parameter.
Signed-off-by: Matthew R. Ochs <mochs@nvidia.com> --- .../pcihole64-virt.aarch64-latest.args | 31 +++++++++++++++++++ .../pcihole64-virt.aarch64-latest.xml | 29 +++++++++++++++++ tests/qemuxmlconfdata/pcihole64-virt.xml | 17 ++++++++++ tests/qemuxmlconftest.c | 1 + 4 files changed, 78 insertions(+) create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.args create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.aarch64-latest.xml create mode 100644 tests/qemuxmlconfdata/pcihole64-virt.xml
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (3)
-
Daniel P. Berrangé -
Matt Ochs -
Matthew R. Ochs