Re: REST service for libvirt to simplify SEV(ES) launch measurement

On Fri, Feb 25, 2022 at 03:10:35PM -0500, Tobin Feldman-Fitzthum wrote: > > > On 2/24/22 7:26 AM, Daniel P. Berrangé wrote: >> On Wed, Feb 23, 2022 at 03:33:22PM -0500, Tobin Feldman-Fitzthum wrote: >>> >>> >>> On 2/23/22 1:38 PM, Dov Murik wrote: >>>> +cc Tobin, James >>>> >>>> On 23/02/2022 19:28, Daniel P. Berrangé wrote: >>>>> Extending management apps using libvirt to support measured launch of >>>>> QEMU guests with SEV/SEV-ES is unreasonably complicated today, both for >>>>> the guest owner and for the cloud management apps. We have APIs for >>>>> exposing info about the SEV host, the SEV guest, guest measurements >>>>> and secret injections. This is a "bags of bits" solution. We expect >>>>> apps to them turn this into a user facting solution. It is possible >>>>> but we're heading to a place where every cloud mgmt app essentially >>>>> needs to reinvent the same wheel and the guest owner will need to >>>>> learn custom APIs for dealing with SEV/SEV-ES for each cloud mgmt >>>>> app. This is pretty awful. We need to do a better job at providing >>>>> a solution that is more general purpose IMHO. >>>>> >> >>> Note in particular that we provide a client script called LaunchVM.py >>> that uses libvirt to start an SEV VM in conjunction with the attestation >>> server. This is basically a stand in for a management app or cloud >>> control plane. The modifications needed to launch an SEV VM are not >>> particularly extensive. I agree with some of your comments though. In >>> some ways it might be nice to have libvirt take care of things and hide >>> the complexity from the management app. >> >> LaunchVM.py nicely illustrates my concerns. Every application that >> uses libvirt that knows how to start VMs, now needs to be changed >> to support the series of operations shown in LaunchVM.py. THe guest >> owner probably can't use LaunchVM.py except for demoware, as they'll >> need a equivalent that talks to the API of their cloud mgmt app, >> of which there are many. >> >>> When we started working on our attestation server, our initial plan was >>> to make PRs to libvirt that would add one end of the attestation API to >>> libvirt, which would directly query the KBS. This is basically what you >>> are proposing. We decided against this for a couple of reasons. >>> >>> First, we were concerned that libvirt might not have network >>> connectivity to an arbitrary attestation server in a cloud environment. >>> We had envisioned that the guest owner would provide a URI for their >>> attestation server as part of the XML. This assumes that the node where >>> the VM is going to run can connect to an attestation server living >>> somewhere on the internet. I think that this might be challenging in >>> some cloud environments. By having the management app connect to libvirt >>> and the attestation server, we add some flexibility. >> >> Agreed, we can't assume that libvirt will always have ability to >> connect to an arbitrary service on the internet. >> >> That said, it does not neccessarily need this ability. If the user >> gives a URL of 'https://myhost.com/attest', the cloud doesn't have >> to give that straight to libvirt. The cloud software could have a >> attestation proxy server. So they could tell libvirt to use the >> URI https://10.0.0.5/attest, and then libvirt connects to that, >> it will proxy the calls through the guest owner's real attestation >> server. > > This might slightly contradict the idea of the management app being out > of the loop, but I guess setting up a proxy isn't very difficult. I > think CSPs already do this kind of thing to enable other features. The difference I see with a proxy approach is that it ought to end up being a dumb transport. It won't have to define any protocol or interpret the data, just blindly pass data back & forth. This is already something often done with VNC where the user connects to a public endpoint exposed by the cloud on its internet boundary, which then forwards the data onto QEMU's real VNC server on the compute host. In the VNC case, the public facing side often does websockets encapsulation, while the private side is pure VNC, but it still doesn't ened to understnd the VNC protocol so it is fairly easy to setup such a proxy. A proxy could also address the other problem I've realized. At least the first VM to be booted on a given cloud might be harder to attest because the attestation service would need to exist outside the cloud being used because the guest owner won't trust anything initially. This could imply that the attestation service is on a local machine controlled by the guest owner, even their local laptop. The implication is that the attestation service could easily be stuck behind NAT and be unable to accept incoming connections from the cloud. To address this the proxy might need to sit in the middle and accept incoming connections from both the guest owner's attestation service and from libvirt on the compute host. IOW, with VNC you get a unidirectional flow of connection establishment guest owner -> cloud proxy -> compute host but with the attestation service you might need a flow with the second arrow reversed ie guest owner -> cloud proxy <- compute host

Once the first VM is bootstrapped, it could be used to run an attestation service that is local to the cloud, avoiding the need for the traffic to traverse the internet when booting future VMs, and removing the dependancy on the guest owner having a local machine to run an attestation service on. >> The needs of all the apps look similar enough, because they are all >> ultimately constrained by the functionality made available by SEV(ES). >> Thus at least wrt apps doing traditional VM management using QEMU, >> it feels like it should be practical to come up with a common solution. >> I am all for common ground here. I had basically given up on it, but > maybe libvirt has enough influence to set a standard. In your first > email you said that there is a relatively short window to come up with > something and I think that is probably correct. Yes, there are a handful of major cloud software projects that use libvirt, and once they've all built their own solution, it won't be so easy to convince them to change into a shared solution, as it would create back compat problems for them. >> I can understand if it is harder to achieve commonality with tech >> like libkrun though, since that's consuming virt in quite a different >> way at the userspace level. > Yeah, extending the focus beyond SEV(-ES) with QEMU might make things > more difficult. There is some discussion right now about trying to find > common ground between SEV-SNP and TDX attestation, but I assume that is > all out of scope since libvirt isn't really involved. I admit I don't know much about TDX, but from what I've understood talking to other people, SEV-SNP might not end up looking all that different. IIUC the attestation has to be initiated from inside the SNP guest after CPUs are running. It is going need to be run as early as possible and while you might be able todo it in the initrd, it feels likely that it could be put into the firmware (OVMF) instead, such that it does the validation before even loading the kernel. This would facilitate supporting it with arbitrary guest OS, as the firmware is common to all. We can't assume the firmware will have direct network connectivity to any attestation service needed to verify the boot. This implies the firmware might need to talk to the host via something like virtio-serial / virtio-vsock, from where libvirt or QEMU can proxy the traffic onto the real attestation service. Such an architecture might end up aligning quite well with SEV/SEV-ES, possible allowing the same protocol to be used in both cases, just with differnt ultimate end points (libvirt for SEV(-ES) vs guest firmware for SEV-SNP).

Regards, Daniel