Re: REST service for libvirt to simplify SEV(ES) launch measurement

On 3/3/22 12:20 PM, Daniel P. Berrangé wrote: > On Fri, Feb 25, 2022 at 03:10:35PM -0500, Tobin Feldman-Fitzthum wrote: >> >> >> On 2/24/22 7:26 AM, Daniel P. Berrangé wrote: >>> On Wed, Feb 23, 2022 at 03:33:22PM -0500, Tobin Feldman-Fitzthum wrote: >>>> >>>> >>>> On 2/23/22 1:38 PM, Dov Murik wrote: >>>>> +cc Tobin, James >>>>> >>>>> On 23/02/2022 19:28, Daniel P. Berrangé wrote: >>>>>> Extending management apps using libvirt to support measured launch of >>>>>> QEMU guests with SEV/SEV-ES is unreasonably complicated today, both for >>>>>> the guest owner and for the cloud management apps. We have APIs for >>>>>> exposing info about the SEV host, the SEV guest, guest measurements >>>>>> and secret injections. This is a "bags of bits" solution. We expect >>>>>> apps to them turn this into a user facting solution. It is possible >>>>>> but we're heading to a place where every cloud mgmt app essentially >>>>>> needs to reinvent the same wheel and the guest owner will need to >>>>>> learn custom APIs for dealing with SEV/SEV-ES for each cloud mgmt >>>>>> app. This is pretty awful. We need to do a better job at providing >>>>>> a solution that is more general purpose IMHO. >>>>>> >>> >>>> Note in particular that we provide a client script called LaunchVM.py >>>> that uses libvirt to start an SEV VM in conjunction with the attestation >>>> server. This is basically a stand in for a management app or cloud >>>> control plane. The modifications needed to launch an SEV VM are not >>>> particularly extensive. I agree with some of your comments though. In >>>> some ways it might be nice to have libvirt take care of things and hide >>>> the complexity from the management app. >>> >>> LaunchVM.py nicely illustrates my concerns. Every application that >>> uses libvirt that knows how to start VMs, now needs to be changed >>> to support the series of operations shown in LaunchVM.py. THe guest >>> owner probably can't use LaunchVM.py except for demoware, as they'll >>> need a equivalent that talks to the API of their cloud mgmt app, >>> of which there are many. >>> >>>> When we started working on our attestation server, our initial plan was >>>> to make PRs to libvirt that would add one end of the attestation API to >>>> libvirt, which would directly query the KBS. This is basically what you >>>> are proposing. We decided against this for a couple of reasons. >>>> >>>> First, we were concerned that libvirt might not have network >>>> connectivity to an arbitrary attestation server in a cloud environment. >>>> We had envisioned that the guest owner would provide a URI for their >>>> attestation server as part of the XML. This assumes that the node where >>>> the VM is going to run can connect to an attestation server living >>>> somewhere on the internet. I think that this might be challenging in >>>> some cloud environments. By having the management app connect to libvirt >>>> and the attestation server, we add some flexibility. >>> >>> Agreed, we can't assume that libvirt will always have ability to >>> connect to an arbitrary service on the internet. >>> >>> That said, it does not neccessarily need this ability. If the user >>> gives a URL of 'https://myhost.com/attest', the cloud doesn't have >>> to give that straight to libvirt. The cloud software could have a >>> attestation proxy server. So they could tell libvirt to use the >>> URI https://10.0.0.5/attest, and then libvirt connects to that, >>> it will proxy the calls through the guest owner's real attestation >>> server. >> >> This might slightly contradict the idea of the management app being out >> of the loop, but I guess setting up a proxy isn't very difficult. I >> think CSPs already do this kind of thing to enable other features. > > The difference I see with a proxy approach is that it ought to end > up being a dumb transport. It won't have to define any protocol or > interpret the data, just blindly pass data back & forth. > > This is already something often done with VNC where the user connects > to a public endpoint exposed by the cloud on its internet boundary, > which then forwards the data onto QEMU's real VNC server on the > compute host. In the VNC case, the public facing side often does > websockets encapsulation, while the private side is pure VNC, but > it still doesn't ened to understnd the VNC protocol so it is fairly > easy to setup such a proxy. > > A proxy could also address the other problem I've realized. At least > the first VM to be booted on a given cloud might be harder to attest > because the attestation service would need to exist outside the cloud > being used because the guest owner won't trust anything initially. > > This could imply that the attestation service is on a local machine > controlled by the guest owner, even their local laptop. The implication > is that the attestation service could easily be stuck behind NAT and > be unable to accept incoming connections from the cloud. > > To address this the proxy might need to sit in the middle and accept > incoming connections from both the guest owner's attestation service > and from libvirt on the compute host. > > IOW, with VNC you get a unidirectional flow of connection > establishment > > guest owner -> cloud proxy -> compute host > > but with the attestation service you might need a flow > with the second arrow reversed ie > > guest owner -> cloud proxy <- compute host > > Yeah I think a proxy is a reasonable solution. I am still not sure where people are going to run their attestation servers in practice. We have been assuming that they could be anywhere. I don't know if people will insist on running them locally or if they will decide to trust some managed attestation solution. I guess we will see. > Once the first VM is bootstrapped, it could be used to run an > attestation service that is local to the cloud, avoiding the > need for the traffic to traverse the internet when booting > future VMs, and removing the dependancy on the guest owner having > a local machine to run an attestation service on. > > >>> The needs of all the apps look similar enough, because they are all >>> ultimately constrained by the functionality made available by SEV(ES). >>> Thus at least wrt apps doing traditional VM management using QEMU, >>> it feels like it should be practical to come up with a common solution. >>> I am all for common ground here. I had basically given up on it, but >> maybe libvirt has enough influence to set a standard. In your first >> email you said that there is a relatively short window to come up with >> something and I think that is probably correct. > > Yes, there are a handful of major cloud software projects that use > libvirt, and once they've all built their own solution, it won't be > so easy to convince them to change into a shared solution, as it > would create back compat problems for them. > >>> I can understand if it is harder to achieve commonality with tech >>> like libkrun though, since that's consuming virt in quite a different >>> way at the userspace level. >> Yeah, extending the focus beyond SEV(-ES) with QEMU might make things >> more difficult. There is some discussion right now about trying to find >> common ground between SEV-SNP and TDX attestation, but I assume that is >> all out of scope since libvirt isn't really involved. > > I admit I don't know much about TDX, but from what I've understood > talking to other people, SEV-SNP might not end up looking all that > different. IIUC the attestation has to be initiated from inside > the SNP guest after CPUs are running. It is going need to be run > as early as possible and while you might be able todo it in the > initrd, it feels likely that it could be put into the firmware (OVMF) > instead, such that it does the validation before even loading the > kernel. This would facilitate supporting it with arbitrary guest OS, > as the firmware is common to all. We can't assume the firmware will > have direct network connectivity to any attestation service needed > to verify the boot. This implies the firmware might need to talk > to the host via something like virtio-serial / virtio-vsock, from > where libvirt or QEMU can proxy the traffic onto the real attestation > service. Such an architecture might end up aligning quite well with > SEV/SEV-ES, possible allowing the same protocol to be used in both > cases, just with differnt ultimate end points (libvirt for SEV(-ES) > vs guest firmware for SEV-SNP). Yeah that is an interesting point. Most SNP approaches that I have seen so far use the kernel/initrd to handle decryption. There is potentially a gap if the kernel/initrd are not themselves part of the measurement that is provided in the attestation report. We have been using this measured direct boot thing for SEV(-ES) and I think it can be extended to SEV-SNP as well. This would close that gap and make it feasible to do the decryption in the kernel.

There might be reasons to do the measurement earlier, however. For instance, it is easier to keep track of the hashes of fewer things (just the firmware vs the firmware + initrd + kernel + cmdline). As you say networking becomes a bit of an issue if you do the attestation in firmware. Using a local device that is handled by libvirt could be a good solution. I'm sure we could come up with a protocol that is general enough to handle both pre-attestation and runtime attestation, but there definitely are some differences. For instance with pre-attestation we know that there will basically be two calls from management app to attestation server. This is defined by the way that we inject secrets. With SNP things are much more flexible. We can setup a persistent connection between the guest and the attestation server and ask for secrets throughout the runtime of the guest. It might take some thinking to reconcile these approaches and could put some dependencies on how the guest is supposed to behave, which isn't really our business (although a firmware-based solution could be reasonable). -Tobin > > Regards, > Daniel