[RFC] Allowing SEV attestation
Dear list, in the light of recent development of secure virtualization (for instance AMD SEV-SNP [1]) I'd like us to be prepared for when QEMU adopts these new technologies and thus would like to discuss our options. So far, I've came across AMD SEV-SNP [2]. While it's true that we do support SEV, we need to adopt its newer version too. Consider the following example: you loan a VM on a public cloud and you want to process some private data there. Of course, the data has to be stored on an encrypted disk/partition. But encrypting guest memory (confidentiality) prevents you only from finding the key in a memory dump. It doesn't help if the guest image, initial guest memory, hypervisor or firmware were tampered with. This is where attestation comes to help - it enables the guest owner (which in this example is different to the one running it) verify - with cryptographic level of confidence - that data confidentiality and integrity can be ensured. Depending on the technology, attestation will be done at different times in the VM lifecycle. When it's done before starting guest vCPUs, it's called pre-attestation and when it's done after the guest has launched, it's called post-attestation. For example, AMD SEV and SEV-ES require pre-attestation. SEV-SNP allows for post attestation. The crux of the issue here is that *when* and *how* the guest owner will interact with the VM for attestation will be different depending on which technology is being used. It looks like QEMU will expose commands needed for attestation via QMP [3]. But question then is, how to expose those at Libvirt level? Should we allow users to bypass Libvirt and communicate to QEMU directly or wrap those QMPs in public APIs, or something completely different? Will those APIs be generic enough to serve other technologies too? For instance, given set of APIs in given order works for SEV but might not work for Intel's TDX. 1: https://kvmforum2019.sched.com/event/Tmwl/improving-and-expanding-sev-suppor... 2: https://www.amd.com/system/files/TechDocs/SEV-SNP-strengthening-vm-isolation... 3: https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg03689.html Michal
On Thu, May 06, 2021 at 12:22:26PM +0200, Michal Prívozník wrote: Hi, (Just chiming in as a curious libvirt API user :-)) [...]
This is where attestation comes to help - it enables the guest owner (which in this example is different to the one running it) verify - with cryptographic level of confidence - that data confidentiality and integrity can be ensured.
Depending on the technology, attestation will be done at different times in the VM lifecycle. When it's done before starting guest vCPUs, it's called pre-attestation and when it's done after the guest has launched, it's called post-attestation. For example, AMD SEV and SEV-ES require pre-attestation. SEV-SNP allows for post attestation. The crux of the issue here is that *when* and *how* the guest owner will interact with the VM for attestation will be different depending on which technology is being used.
It looks like QEMU will expose commands needed for attestation via QMP [3]. But question then is, how to expose those at Libvirt level? Should we allow users to bypass Libvirt and communicate to QEMU directly or wrap those QMPs in public APIs, or something completely different?
I know you're just thinking out loud here, but IIUC, isn't allowing libvirt API users to communicate directly with QEMU in this case departing from the libvirt norm? (Perhaps there's scope of sensible exceptions ... as the devil is in the specs.) Allowing a way to do it is one thing — which is nice for testing and development — but saying "just directly communicate via QMP passthrough" makes the libvirt API user wonder "why is it okay to passthrough QMP commands in this case, but generally such an action is marked as 'tainted' by libvirt?"
Will those APIs be generic enough to serve other technologies too? For instance, given set of APIs in given order works for SEV but might not work for Intel's TDX.
That's a tough question, I guess, on the "API genericness". I have not read the specifictations of AMD and Intel in full; only read them in parts ... but "ideally" (I know), a single set of libvirt APIs would abstract AMD's and Intel's implementation details.
1: https://kvmforum2019.sched.com/event/Tmwl/improving-and-expanding-sev-suppor...
2: https://www.amd.com/system/files/TechDocs/SEV-SNP-strengthening-vm-isolation...
3: https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg03689.html
Michal
-- /kashyap
On 5/6/21 6:35 AM, Kashyap Chamarthy wrote:
It looks like QEMU will expose commands needed for attestation via QMP [3]. But question then is, how to expose those at Libvirt level? Should we allow users to bypass Libvirt and communicate to QEMU directly or wrap those QMPs in public APIs, or something completely different?
I know you're just thinking out loud here, but IIUC, isn't allowing libvirt API users to communicate directly with QEMU in this case departing from the libvirt norm? (Perhaps there's scope of sensible exceptions ... as the devil is in the specs.)
Allowing a way to do it is one thing — which is nice for testing and development — but saying "just directly communicate via QMP passthrough" makes the libvirt API user wonder "why is it okay to passthrough QMP commands in this case, but generally such an action is marked as 'tainted' by libvirt?"
Would this be acceptable if libvirt could apply a policy to the QMP socket which only allowed SEV messages through and dropped all others (just for an example)? If it's possible to leverage the existing QMP API for the attestation messages, then libvirt won't have to duplicate a lower-level attestation protocol. This would allow libvirt to essentially proxy relevant messages and this moves all the attestation complexity to the "edges" (i.e., the client/relying party and the VMM). Connor
On Thu, May 06, 2021 at 07:57:43AM -0500, Connor Kuehl wrote:
On 5/6/21 6:35 AM, Kashyap Chamarthy wrote:
It looks like QEMU will expose commands needed for attestation via QMP [3]. But question then is, how to expose those at Libvirt level? Should we allow users to bypass Libvirt and communicate to QEMU directly or wrap those QMPs in public APIs, or something completely different?
I know you're just thinking out loud here, but IIUC, isn't allowing libvirt API users to communicate directly with QEMU in this case departing from the libvirt norm? (Perhaps there's scope of sensible exceptions ... as the devil is in the specs.)
Allowing a way to do it is one thing — which is nice for testing and development — but saying "just directly communicate via QMP passthrough" makes the libvirt API user wonder "why is it okay to passthrough QMP commands in this case, but generally such an action is marked as 'tainted' by libvirt?"
Would this be acceptable if libvirt could apply a policy to the QMP socket which only allowed SEV messages through and dropped all others (just for an example)? If it's possible to leverage the existing QMP API for the attestation messages, then libvirt won't have to duplicate a lower-level attestation protocol.
Exposing QMP directly to users or higher level tools is not a desirable thing to do. It is not maintained as a long term stable API. QEMU only guarantees it is compatible across 2 QEMU releases, as this is the length of the QEMU deprecation + removal process. This is not something you want to expose to higher level users who will want their tools to be using something that has compatibility on a much longer time frame. Libvirt provides a long term stable API, such that QEMU has the freedom to evolve its low level API on a much faster timeframe than it would otherwise be able to do. As mentioned in my other response, management apps are not likely to want to expose low level hypervisor impl details directly to their users. They typically expose a higher level concept to users that is agnostic to the hypervisor details, to insulate their users from the implementation and specific hypervisor versions.
From a security POV I would also not be happy with untrusted end users having a direct connection to QMP, even if the command set is filtered. It is just one protocol bug away from users having ability to directly exploit the QEMU process.
This would allow libvirt to essentially proxy relevant messages and this moves all the attestation complexity to the "edges" (i.e., the client/relying party and the VMM).
I think need to thinking about exactly what the end user needs to be able to accomplish in general terms, not in terms of low level qemu commands. From there we can understand what an app like OpenStack/KubeVirt needs to enable its users to do, and thus what libvirt needs to providing to support OpenStack/KubeVirt in doing this. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, May 06, 2021 at 12:22:26PM +0200, Michal Prívozník wrote:
Dear list,
in the light of recent development of secure virtualization (for instance AMD SEV-SNP [1]) I'd like us to be prepared for when QEMU adopts these new technologies and thus would like to discuss our options. So far, I've came across AMD SEV-SNP [2]. While it's true that we do support SEV, we need to adopt its newer version too.
Consider the following example: you loan a VM on a public cloud and you want to process some private data there. Of course, the data has to be stored on an encrypted disk/partition. But encrypting guest memory (confidentiality) prevents you only from finding the key in a memory dump. It doesn't help if the guest image, initial guest memory, hypervisor or firmware were tampered with.
This is where attestation comes to help - it enables the guest owner (which in this example is different to the one running it) verify - with cryptographic level of confidence - that data confidentiality and integrity can be ensured.
Depending on the technology, attestation will be done at different times in the VM lifecycle. When it's done before starting guest vCPUs, it's called pre-attestation and when it's done after the guest has launched, it's called post-attestation. For example, AMD SEV and SEV-ES require pre-attestation. SEV-SNP allows for post attestation. The crux of the issue here is that *when* and *how* the guest owner will interact with the VM for attestation will be different depending on which technology is being used.
It looks like QEMU will expose commands needed for attestation via QMP [3].
As mentioned in my reply to that thread, I believe we can already do pretty much all of that via a combination of libvirt APIs & guest XML. The query-sev / query-sev-capabilities info is exposed via the domain capabilities API. The initial startup of the attestation process is doable via the QEMU CLI, and libvirt populates that from info in the domain XML doc. We have an API to query the measurement mapping to query-sev-launch-measure and we hav the ability to start a guest in the paused state, so that it doesn't immediately run guest CPUs. The bit I don't see us implementing is the sev-inject-launch-secret command. I guess we need a new API for that.
But question then is, how to expose those at Libvirt level? Should we allow users to bypass Libvirt and communicate to QEMU directly or wrap those QMPs in public APIs, or something completely different?
Definitely not talk to QEMU directly. That has always been a bad idea for anything, because QEMU is a constantly evolving app, and libvirt needs to provide stability on that moving base.
Will those APIs be generic enough to serve other technologies too? For instance, given set of APIs in given order works for SEV but might not work for Intel's TDX.
We made a half-hearted attempt at being generic in the current APIS and XML we've done so far for SEV, but I'm not especially convinced it will be good enough for TDX too. Time will tell. Given that we have most of the stuff for SEV already implemented, we've already accepted that risk, so finishing the job for SEV isn't making the situation significantly worse for us.
1: https://kvmforum2019.sched.com/event/Tmwl/improving-and-expanding-sev-suppor...
2: https://www.amd.com/system/files/TechDocs/SEV-SNP-strengthening-vm-isolation...
3: https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg03689.html
Michal
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 5/6/21 6:51 AM, Daniel P. Berrangé wrote:
It looks like QEMU will expose commands needed for attestation via QMP [3].
As mentioned in my reply to that thread, I believe we can already do pretty much all of that via a combination of libvirt APIs & guest XML.
This is not a good user experience. The entire attestation process should be made ephemeral, taking place 100% over a socket. Enabling a fully socket-based attestation workflow will decouple it from the domain XML and the host file system and make it easier for guest-owner tooling to facilitate attestation. Connor
On Thu, May 06, 2021 at 08:04:44AM -0500, Connor Kuehl wrote:
On 5/6/21 6:51 AM, Daniel P. Berrangé wrote:
It looks like QEMU will expose commands needed for attestation via QMP [3].
As mentioned in my reply to that thread, I believe we can already do pretty much all of that via a combination of libvirt APIs & guest XML.
This is not a good user experience. The entire attestation process should be made ephemeral, taking place 100% over a socket. Enabling a fully socket-based attestation workflow will decouple it from the domain XML and the host file system and make it easier for guest-owner tooling to facilitate attestation.
The libvirt library is also just a client talking to a socket, so concept that's no different, just using a protocol libvirt has defined at a higher level instead of a low level hypervisor specific protocol. The tools that are managing the VM lifecycle are already using libvirt as their API, and already doing several of the steps described wrt to SEV. It is not realistic to expect them to take a completely different approach to managing the startup of VMs that have SEV enabled, they need to have a way to integrate with their existing approach to mgmt. The guest owner has to in turn talk to some mechanism that the management app exposes to them. These management apps are pretty unlikely to wish to directly expose a low level impl detail of QEMU. They won't even expose the fact that they're using libvirt in fact, instead presenting some higher level API again, as they rarely wish to leak low level hypervisor details outside as that locks them into supporting specific low level details long term. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 5/6/21 8:32 AM, Daniel P. Berrangé wrote:
On Thu, May 06, 2021 at 08:04:44AM -0500, Connor Kuehl wrote:
On 5/6/21 6:51 AM, Daniel P. Berrangé wrote:
It looks like QEMU will expose commands needed for attestation via QMP [3].
As mentioned in my reply to that thread, I believe we can already do pretty much all of that via a combination of libvirt APIs & guest XML.
This is not a good user experience. The entire attestation process should be made ephemeral, taking place 100% over a socket. Enabling a fully socket-based attestation workflow will decouple it from the domain XML and the host file system and make it easier for guest-owner tooling to facilitate attestation.
The libvirt library is also just a client talking to a socket, so concept that's no different, just using a protocol libvirt has defined at a higher level instead of a low level hypervisor specific protocol.
The tools that are managing the VM lifecycle are already using libvirt as their API, and already doing several of the steps described wrt to SEV. It is not realistic to expect them to take a completely different approach to managing the startup of VMs that have SEV enabled, they need to have a way to integrate with their existing approach to mgmt.
The guest owner has to in turn talk to some mechanism that the management app exposes to them. These management apps are pretty unlikely to wish to directly expose a low level impl detail of QEMU. They won't even expose the fact that they're using libvirt in fact, instead presenting some higher level API again, as they rarely wish to leak low level hypervisor details outside as that locks them into supporting specific low level details long term.
I see. So it sounds like the way forward for libvirt is that it will need to essentially duplicate the SEV-related QMP message types into its own protocol since expecting the client to understand QMP discloses the fact that the underlying hypervisor is QEMU. Connor
On Thu, May 06, 2021 at 08:43:53AM -0500, Connor Kuehl wrote:
On 5/6/21 8:32 AM, Daniel P. Berrangé wrote:
On Thu, May 06, 2021 at 08:04:44AM -0500, Connor Kuehl wrote:
On 5/6/21 6:51 AM, Daniel P. Berrangé wrote:
It looks like QEMU will expose commands needed for attestation via QMP [3].
As mentioned in my reply to that thread, I believe we can already do pretty much all of that via a combination of libvirt APIs & guest XML.
This is not a good user experience. The entire attestation process should be made ephemeral, taking place 100% over a socket. Enabling a fully socket-based attestation workflow will decouple it from the domain XML and the host file system and make it easier for guest-owner tooling to facilitate attestation.
The libvirt library is also just a client talking to a socket, so concept that's no different, just using a protocol libvirt has defined at a higher level instead of a low level hypervisor specific protocol.
The tools that are managing the VM lifecycle are already using libvirt as their API, and already doing several of the steps described wrt to SEV. It is not realistic to expect them to take a completely different approach to managing the startup of VMs that have SEV enabled, they need to have a way to integrate with their existing approach to mgmt.
The guest owner has to in turn talk to some mechanism that the management app exposes to them. These management apps are pretty unlikely to wish to directly expose a low level impl detail of QEMU. They won't even expose the fact that they're using libvirt in fact, instead presenting some higher level API again, as they rarely wish to leak low level hypervisor details outside as that locks them into supporting specific low level details long term.
I see. So it sounds like the way forward for libvirt is that it will need to essentially duplicate the SEV-related QMP message types into its own protocol since expecting the client to understand QMP discloses the fact that the underlying hypervisor is QEMU.
I wouldn't say duplicate the SEV QMP messages, as I wouldn't assume that there's a direct 1:1 mapping between what libvirt exposes and what QMP exposes. There will be some mapping, but it could well be M:N, since libvirt aims to express things as higher level concepts that stand a better chance of being cross-hypervisor applicable [1], rather than just 1:1 duplicating QEMU. Regards, Daniel [1] In practice we might not achieve hypervisor portability, especially if QEMU is the only impl that exists, as we can't reliably predict the future of what other hypervisors may do for the same concept. -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 5/6/21 8:51 AM, Daniel P. Berrangé wrote:
I see. So it sounds like the way forward for libvirt is that it will need to essentially duplicate the SEV-related QMP message types into its own protocol since expecting the client to understand QMP discloses the fact that the underlying hypervisor is QEMU.
I wouldn't say duplicate the SEV QMP messages, as I wouldn't assume that there's a direct 1:1 mapping between what libvirt exposes and what QMP exposes. There will be some mapping, but it could well be M:N, since libvirt aims to express things as higher level concepts that stand a better chance of being cross-hypervisor applicable [1], rather than just 1:1 duplicating QEMU.
The SEV QMP messages are just JSON representations of the SEV firmware data structures. The client requires exactly those data structures and nothing more; so it is almost certainly and unavoidably going to be a 1:1 cut and paste job. So to be clear, it's not duplicating QEMU, just unpacking from QMP and dropping the data structures into whatever message type libvirt exposes. Connor
On 5/6/21 04:22, Michal Prívozník wrote:
Dear list,
Hi Michal, This thread has been quiet for a long time, but I wanted to check if any work has been done to provide an sev-inject-launch-secret equivalent for libvirt. AFAICT, there was agreement this missing piece is needed to solve the attestation puzzle. Did you make any progress? If so, I can help with testing and review. If not, I can take a stab at it. Regards, Jim
in the light of recent development of secure virtualization (for instance AMD SEV-SNP [1]) I'd like us to be prepared for when QEMU adopts these new technologies and thus would like to discuss our options. So far, I've came across AMD SEV-SNP [2]. While it's true that we do support SEV, we need to adopt its newer version too.
Consider the following example: you loan a VM on a public cloud and you want to process some private data there. Of course, the data has to be stored on an encrypted disk/partition. But encrypting guest memory (confidentiality) prevents you only from finding the key in a memory dump. It doesn't help if the guest image, initial guest memory, hypervisor or firmware were tampered with.
This is where attestation comes to help - it enables the guest owner (which in this example is different to the one running it) verify - with cryptographic level of confidence - that data confidentiality and integrity can be ensured.
Depending on the technology, attestation will be done at different times in the VM lifecycle. When it's done before starting guest vCPUs, it's called pre-attestation and when it's done after the guest has launched, it's called post-attestation. For example, AMD SEV and SEV-ES require pre-attestation. SEV-SNP allows for post attestation. The crux of the issue here is that *when* and *how* the guest owner will interact with the VM for attestation will be different depending on which technology is being used.
It looks like QEMU will expose commands needed for attestation via QMP [3]. But question then is, how to expose those at Libvirt level? Should we allow users to bypass Libvirt and communicate to QEMU directly or wrap those QMPs in public APIs, or something completely different?
Will those APIs be generic enough to serve other technologies too? For instance, given set of APIs in given order works for SEV but might not work for Intel's TDX.
1: https://kvmforum2019.sched.com/event/Tmwl/improving-and-expanding-sev-suppor...
2: https://www.amd.com/system/files/TechDocs/SEV-SNP-strengthening-vm-isolation...
3: https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg03689.html
Michal
On 10/27/21 1:29 AM, Jim Fehlig wrote:
On 5/6/21 04:22, Michal Prívozník wrote:
Dear list,
Hi Michal,
This thread has been quiet for a long time, but I wanted to check if any work has been done to provide an sev-inject-launch-secret equivalent for libvirt. AFAICT, there was agreement this missing piece is needed to solve the attestation puzzle. Did you make any progress? If so, I can help with testing and review. If not, I can take a stab at it.
Please, be my guest. I don't have capacity to do that myself currently, but I might have some to do review. Thanks, Michal
On Tue, Oct 26, 2021 at 05:29:00PM -0600, Jim Fehlig wrote:
On 5/6/21 04:22, Michal Prívozník wrote:
Dear list,
Hi Michal,
This thread has been quiet for a long time, but I wanted to check if any work has been done to provide an sev-inject-launch-secret equivalent for libvirt. AFAICT, there was agreement this missing piece is needed to solve the attestation puzzle. Did you make any progress? If so, I can help with testing and review. If not, I can take a stab at it.
I've not started any work, but was thinking about it a little, but not much further than considering that we should have an API that looks like int virDomainSetLaunchSecurityInfo(virDomainPtr domain, virTypedParameterPtr *params, int *nparams, unsigned int flags); though this is little unusual because in other APIs where we have Set and Get functions for virTypedParameterPtr, we allow the same set of typed parameter keys for the Set/Get. In this case we would have different parameters for the Set/Get scenarios So I was thinking perhaps we should just a different name for the setter but then failed to come up with a nice alternative. Naming is always the hardest problem :-) The implemntation of any API would be quite straighforward. Anyway if you have cycles to work on it great, but I can work on it sometime reasonably soon too if you don't. My main constraint is getting acess to hardware to test with... Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 10/27/21 10:11, Daniel P. Berrangé wrote:
On Tue, Oct 26, 2021 at 05:29:00PM -0600, Jim Fehlig wrote:
On 5/6/21 04:22, Michal Prívozník wrote:
Dear list,
Hi Michal,
This thread has been quiet for a long time, but I wanted to check if any work has been done to provide an sev-inject-launch-secret equivalent for libvirt. AFAICT, there was agreement this missing piece is needed to solve the attestation puzzle. Did you make any progress? If so, I can help with testing and review. If not, I can take a stab at it.
I've not started any work, but was thinking about it a little, but not much further than considering that we should have an API that looks like
int virDomainSetLaunchSecurityInfo(virDomainPtr domain, virTypedParameterPtr *params, int *nparams, unsigned int flags);
though this is little unusual because in other APIs where we have Set and Get functions for virTypedParameterPtr, we allow the same set of typed parameter keys for the Set/Get. In this case we would have different parameters for the Set/Get scenarios
So I was thinking perhaps we should just a different name for the setter but then failed to come up with a nice alternative.
Naming is always the hardest problem :-) The implemntation of any API would be quite straighforward.
Naming can be hard, sometimes more so than the coding :-). To provide some asymmetry, how about virDomainInjectLaunchSecurityInfo?
Anyway if you have cycles to work on it great, but I can work on it sometime reasonably soon too if you don't. My main constraint is getting acess to hardware to test with...
I'll be on vacation until Nov 9 and can work on it when returning. If you happen to have time before then, I'd be happy to review and test. I do have shared access to test hardware. Regards, Jim
participants (5)
-
Connor Kuehl -
Daniel P. Berrangé -
Jim Fehlig -
Kashyap Chamarthy -
Michal Prívozník