Hi All, (sorry for the long message) CPUs from AMD EPYC family supports Secure Encrypted Virtualization (SEV) feature - the feature allows running encrypted VMs. To enable the feature, I have been submitting patches to Linux kernel [1], Qemu [2] and OVMF [3]. We have been making some good progress in getting patches accepted upstream in Linux and OVMF trees. SEV builds upon SME (Secure Memory Encryption) feature -- SME support just got pulled into 4.14 merge window. The base SEV patches are accepted in OVMF tree -- now we have SEV aware guest BIOS. I am getting ready to take off "RFC" tag from remaining patches to get them reviewed and accepted. The boot flow for launching an SEV guest is a bit different from a typical guest launch. In order to launch SEV guest from virt-manager or other high-level VM management tools, we need to design and implement new interface between libvirt and qemu, and probably add new APIs in libvirt to be used by VM management tools. I am new to the libvirt and need some expert advice while designing this interface. A pictorial representation for a SEV guest launch flow is available in SEV Spec Appendix A [4]. A typical flow looks like this: 1. Guest owner (GO) asks the cloud provider to launch SEV guest. 2. VM tool asks libvirt to provide its Platform Diffie-Hellman (PDH) key. 3. libvirt opens /dev/sev device to get its PDH and return the blob to the caller.

4. VM tool gives its PDH to GO. 5. GO provides its DH key, session-info and guest policy.

6. VM tool somehow communicates the GO provided information to libvirt.

7. libvirt adds "sev-guest" object in its xml file with all the information obtained from #5 (currently my xml file looks like this) <qemu:arg value='-object'> <qemu:arg value='sev-guest,id=sev0,policy=<GO_policy>,dh-key-file=<filename>,session-file=<filename>/> <qemu:arg value='-machine'/> <qemu:arg value='memory-encryption=sev0'/> 8. libvirt launches the guest with "-S"

9. While creating the SEV guest qemu does the following i) create encryption context using GO's DH, session-info and guest policy (LAUNCH_START) ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement 10. By some interface we must propagate the measurement all the way to GO before libvirt starts the guest.

11. GO verifies the measurement and if measurement matches then it may give a secret blob -- which must be injected into the guest before libvirt starts the VM. If verification failed, GO will request cloud provider to destroy the VM.

12. After secret blob is injected into guest, we call LAUNCH_FINISH to destory the encryption context. 13. libvirt issues "continue" command to resume the guest boot.

Please note that the measurement value is protected with transport encryption key (TIK) and it changes on each run. Similarly the secret blob provided by GO does not need to be protected using libvirt/qemu APIs. The secret is protected by TIK. From qemu and libvirt point of view these are blobs and must be passed as-is to the SEV FW. Questions: a) Do we need to add a new set of APIs in libvirt to return the PDH from libvirt and VM tool ? Or can we use some pre-existing APIs to pass the opaque blobs ? (this is mainly for step 3 and 6) b) do we need to define a new xml tag to for memory-encryption ? or just use the qemu:args tag ? (step 6)

c) what existing communicate interface can be used between libvirt and qemu to get the measurement ? can we add a new qemu monitor command 'get_sev_measurement' to get the measurement ? (step 10)

d) how to pass the secret blob from libvirt to qemu ? should we consider adding a new object (sev-guest-secret) -- libvirt can add the object through qemu monitor.

[1] https://marc.info/?l=kvm&m=150092661105069&w=2 [2] https://marc.info/?l=qemu-devel&m=148901186615642&w=2 [3] https://lists.01.org/pipermail/edk2-devel/2017-July/012220.html [4] http://support.amd.com/TechDocs/55766_SEV-KM%20API_Specification.pdf

On Fri, Sep 08, 2017 at 01:45:06PM +0000, Relph, Richard wrote: > A few answers in line… > > On 9/8/17, 8:16 AM, "Daniel P. Berrange" <berrange(a)redhat.com&gt; wrote: > > On Fri, Sep 08, 2017 at 06:57:30AM -0500, Brijesh Singh wrote: > > Hi All, > > > > (sorry for the long message) > > > > CPUs from AMD EPYC family supports Secure Encrypted Virtualization (SEV) > > feature - the feature allows running encrypted VMs. To enable the feature, > > I have been submitting patches to Linux kernel [1], Qemu [2] and OVMF [3]. > > We have been making some good progress in getting patches accepted upstream > > in Linux and OVMF trees. SEV builds upon SME (Secure Memory Encryption) > > feature -- SME support just got pulled into 4.14 merge window. The base > > SEV patches are accepted in OVMF tree -- now we have SEV aware guest BIOS. > > I am getting ready to take off "RFC" tag from remaining patches to get them > > reviewed and accepted. > > > > The boot flow for launching an SEV guest is a bit different from a typical > > guest launch. In order to launch SEV guest from virt-manager or other > > high-level VM management tools, we need to design and implement new > > interface between libvirt and qemu, and probably add new APIs in libvirt > > to be used by VM management tools. I am new to the libvirt and need some > > expert advice while designing this interface. A pictorial representation > > for a SEV guest launch flow is available in SEV Spec Appendix A [4]. > > > > A typical flow looks like this: > > > > 1. Guest owner (GO) asks the cloud provider to launch SEV guest. > > 2. VM tool asks libvirt to provide its Platform Diffie-Hellman (PDH) key. > > 3. libvirt opens /dev/sev device to get its PDH and return the blob to the > > caller. > > What sort of size are we talking about for the PDH ? > > The PDH blob is described in reference 4. It’s 0x824 bytes long… a bit over 2K bytes. > PDH is “Platform Diffie-Hellman” key, public portion. > > There's a few ways libvirt could report it > > 1. As an XML element in the host capabilities XML > 2. As an XML element in the emulator capabilities XML > 3. Via a newly added host API > > > 4. VM tool gives its PDH to GO. > > 5. GO provides its DH key, session-info and guest policy. > > Are steps 4 & 5 strictly required to be in this order, or is it > possible > > Steps 4 and 5 must occur in that order. The data sent by the GO in the > “session info” is encrypted and integrity protected with keys that the > GO derives from the GO private Diffie-Hellman key and the PDH public > Diffie-Hellman key. > > What are the security requirements around the DH key, session info > and guest policy ? Are any of them sensitive data which needs to be > kept private from untrustworthy users. Also are all three of these > items different for every guest launched, or are some of them > likely to be the same for every guest ? > > The PDH is not sensitive. It is relatively static for the platform. > (It only changes when the platform owner chooses to change the apparent > identity of the platform that will actually run the virtual machine.) > The 128-byte session info data is encrypted and integrity protected by > the GO. It need not be additionally encrypted or integrity protected. > It should vary for EVERY guest launch. > The 4-byte guest policy must be sent in the clear as some components > may want to observe the policy bits. It may change from guest to guest, > but there will likely only be a few common values. Given this, and the pretty small data sizes, I think this info could in fact all be provided inline in the XML config - either hex or base64 encoded for the binary blobs. > > 8. libvirt launches the guest with "-S" > > All libvirt guests get launched with -S, to give libvirt chance to do some > setup before starting vCPUs. Normally vCPUs are started by default, but > the VIR_DOMAIN_START_PAUSED flag allows the mgmt app to tell libvirt to > leave vCPUS paused. > > Alternatively, if libvirt sees presencese of 'sev' config for the guest, > it could automatically leave it in PAUSED state regardless of the > VIR_DOMAIN_START_PAUSED flag. > > While using the LAUNCH_MEASURE and LAUNCH_SECRET operations is expected > to be the common case, they are optional. I would recommend requiring > the upstream software to explicitly set the VIR_DOMAIN_START_PAUSED flag, > if only to minimize the dependencies… Ok. > > 9. While creating the SEV guest qemu does the following > > i) create encryption context using GO's DH, session-info and guest policy > > (LAUNCH_START) > > ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) > > iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement > > 10. By some interface we must propagate the measurement all the way to GO > > before libvirt starts the guest. > > Again, what kind of size data are we talking about for athe "measurement" > blob ? a KB, 10's of KB, or more ? > > The measurement is 48 bytes… For that small size we can definitely provide is inline in the event payload then. > My first gut instinct would be for QEMU to emit a QMP event when it has > the measurement available. The event could include the actual data blob, > or we can could add an explicit QMP command to fetch the data blob. > > Libvirt could listen for this QEMU event, and in turn emit an event from > libvirt with the same data, which the mgmt tool can finally give to the > GO. > > > 11. GO verifies the measurement and if measurement matches then it may > > give a secret blob -- which must be injected into the guest before > > libvirt starts the VM. If verification failed, GO will request cloud > > provider to destroy the VM. > > So we need some mechanism to provide the secret blob. This could be > done via a new libvirt API. Alternatively, if we're using virSecret > for the other stuff, the guest config XML could include the UUID of > a 4th secret. Libvirt would then watch to see when that secret has > a value set, and pass that onto QEMU. > > The secret is optional… it is up to 16KB, already encrypted, and integrity > protected with an IV and MAC value passed with the secret. The guest > address the secret should be deposited at is > Technically, the SEV FW API allows there to be more than one secret… > I don’t see a strong reason to require libvirt to support more than one, > though. So I think we could simply add a new API to libvirt to provide this data item. > > 12. After secret blob is injected into guest, we call LAUNCH_FINISH > > to destory the encryption context. > > 13. libvirt issues "continue" command to resume the guest boot. > > This is as simple as the mgmt tool calling virDomainResume to unpause > CPUs. We could have it such that virDomainResume checks whether the > virSecret has been populated secret blob by GO, and pass it onto > QEMU at this time. > > > Please note that the measurement value is protected with transport > > encryption key (TIK) and it changes on each run. Similarly the secret blob > > provided by GO does not need to be protected using libvirt/qemu APIs. The > > secret is protected by TIK. From qemu and libvirt point of view these are > > blobs and must be passed as-is to the SEV FW. > > > > Questions: > > a) Do we need to add a new set of APIs in libvirt to return the PDH from > > libvirt and VM tool ? Or can we use some pre-existing APIs to pass the > > opaque blobs ? (this is mainly for step 3 and 6) > > b) do we need to define a new xml tag to for memory-encryption ? or just > > use the qemu:args tag ? (step 6) > > <qemu:args> is explicitly only ever for ad-hoc testing. > > For anything that is to be used in production deployment we must > explicitly model it in the XML. So we definitely need new XML > defined. > > > c) what existing communicate interface can be used between libvirt and qemu > > to get the measurement ? can we add a new qemu monitor command > > 'get_sev_measurement' to get the measurement ? (step 10) > > Yes, QMP commands seeem most likely. > > > d) how to pass the secret blob from libvirt to qemu ? should we consider > > adding a new object (sev-guest-secret) -- libvirt can add the object through > > qemu monitor. > > Yeah, that looks like a viable option too. So I could see a flow like the following:

1. mgmt tool calls virConnectGetCapabilities. This returns an XML document that includes the following <host> ...other bits... <sev> <platform-key>...hex encoded PDH key...</platform-key> </sev> </host> 2. mgmt tool requests to start a guest calling virCreateXML(), passing VIR_DOMAIN_START_PAUSED. The XML would include <sev> <owner-key>...hex encode DH key...</owner-key> <session-info>..hex encode info...</session-info> <policy>...int32 value..</policy> </sev> if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, libvirt would report an error and refuse to start the guest

3. Libvirt generates the QEMU cli arg to enable SEV using the XML data and starts QEMU, leaving CPUs paused

4. QEMU emits a SEV_MEASURE event containing the measurement blob 5. Libvirt catches the QEMU event and emits its own VIR_CONNECT_DOMAIN_EVENT_SEV_MEASURE event containing the measurement blob 6. GO does its validation of the measurement 7a If validation failed, then virDomainDestroy() to stop QEMU 7b If validation succeeed Optionally call virDomainSetSEVSecret() providing the optional secret, then virDomainResume() to let QEMU continue Regards, Daniel

On Fri, Sep 08, 2017 at 10:48:10AM -0500, Brijesh Singh wrote: >> So I could see a flow like the following: > > > The flow looks good > >> >> >> 1. mgmt tool calls virConnectGetCapabilities. This returns an XML >> document that includes the following >> >> <host> >> ...other bits... >> <sev> >> <platform-key>...hex encoded PDH key...</platform-key> >> </sev> >> </host> >> >> 2. mgmt tool requests to start a guest calling virCreateXML(), >> passing VIR_DOMAIN_START_PAUSED. The XML would include >> >> <sev> >> <owner-key>...hex encode DH key...</owner-key> >> <session-info>..hex encode info...</session-info> >> <policy>...int32 value..</policy> >> </sev> >> >> >> if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, >> libvirt would report an error and refuse to start the guest >> > > > One thing which is not clear to me is, how do we know that we are asked > to launch SEV guest? Are you thinking that <sev> tag in the XML will > hint libvirt that GO has asked to launch a SEV guest? Yes, the existance of the <sev> tag is the indicator that informs libvirt that SEV *must* be used for the guest.

>> 3. Libvirt generates the QEMU cli arg to enable SEV using >> the XML data and starts QEMU, leaving CPUs paused >> > > > I am looking at [1] to get the feel for how do we model it in the XML. > As you can see I am using ad-hoc <qemu:args> to create the sev-guest > object. Currently, sev-guest object accepts the following properties: > > dh-cert-file: <file containing the GO DH key> > session-info-file: <file contain the GO session info> > policy: <int32 GO policy> > > I believe the new XML model will influence the property input type, > Any recommendation on how do model this part ? thank you so much. That looks ok to me - even if QEMU wants the data provided in files on disk, libvirt can just create the files on the fly from the data it has in the <sev> element in the XML file. Since they're only needed during startup, libvirt can then easily delete the files the moment QEMU has completed its startup.

> > > 11. GO verifies the measurement and if measurement matches then it may > > > give a secret blob -- which must be injected into the guest before > > > libvirt starts the VM. If verification failed, GO will request cloud > > > provider to destroy the VM.

* Michael S. Tsirkin (mst(a)redhat.com) wrote: > On Fri, Sep 08, 2017 at 10:48:10AM -0500, Brijesh Singh wrote: > > > > > 11. GO verifies the measurement and if measurement matches then it may > > > > > give a secret blob -- which must be injected into the guest before > > > > > libvirt starts the VM. If verification failed, GO will request cloud > > > > > provider to destroy the VM. > > I realised I'm missing something here: how does GO limit the > secret to the specific VM? For example, what prevents hypervisor > from launching two VMs with the same GO's DH, getting measurement > from 1st one but injecting the secret into the second one? Isn't that the 'trusted channel nonce currently associated with the guest' in the guest context? Dave

> Thanks, > > -- > MST > -- Dr. David Alan Gilbert / dgilbert(a)redhat.com / Manchester, UK

> > > c) what existing communicate interface can be used between libvirt and qemu > > to get the measurement ? can we add a new qemu monitor command > > 'get_sev_measurement' to get the measurement ? (step 10) > > Yes, QMP commands seeem most likely. > > > d) how to pass the secret blob from libvirt to qemu ? should we consider > > adding a new object (sev-guest-secret) -- libvirt can add the object through > > qemu monitor. > > Yeah, that looks like a viable option too. So I could see a flow like the following: 1. mgmt tool calls virConnectGetCapabilities. This returns an XML document that includes the following <host> ...other bits... <sev> <platform-key>...hex encoded PDH key...</platform-key> </sev> </host> 2. mgmt tool requests to start a guest calling virCreateXML(), passing VIR_DOMAIN_START_PAUSED. The XML would include <sev> <owner-key>...hex encode DH key...</owner-key> <session-info>..hex encode info...</session-info> <policy>...int32 value..</policy> </sev> if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, libvirt would report an error and refuse to start the guest 3. Libvirt generates the QEMU cli arg to enable SEV using the XML data and starts QEMU, leaving CPUs paused 4. QEMU emits a SEV_MEASURE event containing the measurement blob

5. Libvirt catches the QEMU event and emits its own VIR_CONNECT_DOMAIN_EVENT_SEV_MEASURE event containing the measurement blob 6. GO does its validation of the measurement 7a If validation failed, then virDomainDestroy() to stop QEMU 7b If validation succeeed Optionally call virDomainSetSEVSecret()

On Mon, Sep 18, 2017 at 11:43:57AM +0200, Erik Skultety wrote: > [...] > >>> > c) what existing communicate interface can be used between libvirt and qemu >>> > to get the measurement ? can we add a new qemu monitor command >>> > 'get_sev_measurement' to get the measurement ? (step 10) >>> >>> Yes, QMP commands seeem most likely. >>> >>> > d) how to pass the secret blob from libvirt to qemu ? should we consider >>> > adding a new object (sev-guest-secret) -- libvirt can add the object through >>> > qemu monitor. >>> >>> Yeah, that looks like a viable option too. >> So I could see a flow like the following: >> >> >> 1. mgmt tool calls virConnectGetCapabilities. This returns an XML >> document that includes the following >> >> <host> >> ...other bits... >> <sev> >> <platform-key>...hex encoded PDH key...</platform-key> >> </sev> >> </host> >> >> 2. mgmt tool requests to start a guest calling virCreateXML(), >> passing VIR_DOMAIN_START_PAUSED. The XML would include >> >> <sev> >> <owner-key>...hex encode DH key...</owner-key> >> <session-info>..hex encode info...</session-info> >> <policy>...int32 value..</policy> >> </sev> >> >> >> if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, >> libvirt would report an error and refuse to start the guest

>> >> 3. Libvirt generates the QEMU cli arg to enable SEV using >> the XML data and starts QEMU, leaving CPUs paused >> >> 4. QEMU emits a SEV_MEASURE event containing the measurement >> blob > Speaking of which, I expect QEMU to have a QMP command to retrieve the > measurement, in which case I think libvirt has to provide an API for the user > to retrieve the measurement in case libvirtd crashes somewhere between setting > up QEMU and waiting for the measurement event from QEMU, or simply because the > GO missed the event for some unspecified reason. Yeah, that's a good point - we also ought to have a pause-reason that reflects that it is paused due to waiting for SEV secrets. >> 5. Libvirt catches the QEMU event and emits its own >> VIR_CONNECT_DOMAIN_EVENT_SEV_MEASURE event containing >> the measurement blob >> >> 6. GO does its validation of the measurement >> >> 7a If validation failed, then virDomainDestroy() to stop QEMU >> >> 7b If validation succeeed >> >> Optionally call >> >> virDomainSetSEVSecret() > Given the fact that we're likely introducing a new <sev> element to the XML > config, I'm more inclined to utilizing the existing virSecret interfaces (as > was originally suggested) instead of creating a vendor-specific API. You could > have an optional secret sub-element within the <sev> element and libvirt would > simply check if that secret has a value set, once the GO issues > virDomainResume(). Any particular reason for having a specific API for this that > I'm missing? Initially I was intending to suggest extensive use of virSecret, but it turns out that despite being called a "secret", none of the SEV data we are passing around needs protection. Either it is safe to be public, or it is already encrypted. So essentially we just have some data blobs we need to pass into QEMU. I didn't feel we ought to be abusing virSecret as a general purpose mechanism for passing in opaque data blobs which do not need any kind of protection.

Regards, Daniel

Hi All,

(sorry for the long message) CPUs from AMD EPYC family supports Secure Encrypted Virtualization (SEV) feature - the feature allows running encrypted VMs. To enable the feature, I have been submitting patches to Linux kernel [1], Qemu [2] and OVMF [3]. We have been making some good progress in getting patches accepted upstream in Linux and OVMF trees. SEV builds upon SME (Secure Memory Encryption) feature -- SME support just got pulled into 4.14 merge window. The base SEV patches are accepted in OVMF tree -- now we have SEV aware guest BIOS. I am getting ready to take off "RFC" tag from remaining patches to get them reviewed and accepted. The boot flow for launching an SEV guest is a bit different from a typical guest launch. In order to launch SEV guest from virt-manager or other high-level VM management tools, we need to design and implement new interface between libvirt and qemu, and probably add new APIs in libvirt to be used by VM management tools. I am new to the libvirt and need some expert advice while designing this interface. A pictorial representation for a SEV guest launch flow is available in SEV Spec Appendix A [4]. A typical flow looks like this: 1. Guest owner (GO) asks the cloud provider to launch SEV guest. 2. VM tool asks libvirt to provide its Platform Diffie-Hellman (PDH) key. 3. libvirt opens /dev/sev device to get its PDH and return the blob to the    caller. 4. VM tool gives its PDH to GO. 5. GO provides its DH key, session-info and guest policy. 6. VM tool somehow communicates the GO provided information to libvirt. 7. libvirt adds "sev-guest" object in its xml file with all the information    obtained from #5    (currently my xml file looks like this)    <qemu:arg value='-object'>    <qemu:arg value='sev-guest,id=sev0,policy=<GO_policy>,dh-key-file=<filename>,session-file=<filename>/>    <qemu:arg value='-machine'/>    <qemu:arg value='memory-encryption=sev0'/> 8. libvirt launches the guest with "-S" 9. While creating the SEV guest qemu does the following  i) create encryption context using GO's DH, session-info and guest policy     (LAUNCH_START)  ii) encrypts the guest bios (LAUNCH_UPDATE_DATA)  iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement

10. By some interface we must propagate the measurement all the way to GO   before libvirt starts the guest. 11. GO verifies the measurement and if measurement matches then it may  give a secret blob -- which must be injected into the guest before  libvirt starts the VM. If verification failed, GO will request cloud  provider to destroy the VM. 12. After secret blob is injected into guest, we call LAUNCH_FINISH   to destory the encryption context. 13. libvirt issues "continue" command to resume the guest boot. Please note that the measurement value is protected with transport encryption key (TIK) and it changes on each run. Similarly the secret blob provided by GO does not need to be protected using libvirt/qemu APIs. The secret is protected by TIK. From qemu and libvirt point of view these are blobs and must be passed as-is to the SEV FW.

Questions: a) Do we need to add a new set of APIs in libvirt to return the PDH from libvirt and VM tool ? Or can we use some pre-existing APIs to pass the opaque blobs ? (this is mainly for step 3 and 6) b) do we need to define a new xml tag to for memory-encryption ? or just use the qemu:args tag ? (step 6) c) what existing communicate interface can be used between libvirt and qemu to get the measurement ? can we add a new qemu monitor command 'get_sev_measurement' to get the measurement ? (step 10) d) how to pass the secret blob from libvirt to qemu ? should we consider adding a new object (sev-guest-secret) -- libvirt can add the object through qemu monitor. [1] https://marc.info/?l=kvm&m=150092661105069&w=2 [2] https://marc.info/?l=qemu-devel&m=148901186615642&w=2 [3] https://lists.01.org/pipermail/edk2-devel/2017-July/012220.html [4] http://support.amd.com/TechDocs/55766_SEV-KM%20API_Specification.pdf Thanks Brijesh

> 8. libvirt launches the guest with "-S" > 9. While creating the SEV guest qemu does the following > i) create encryption context using GO's DH, session-info and guest policy > (LAUNCH_START) > ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) > iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement This part troubles me. This seems to mean that the guest being launched must know what the measurement of the bios is going to be. This means that the cloud provider can not update the bios without breaking guests. Also, while in practice you typically can run an old bios image on a new qemu instance, this is not really tested so would be very hard to support properly in QEMU.

And this looks like a fundamental problem with the hash based measurement that's in hardware. So below I suggest that we layer some software on top to rely on the hash as little as possible. > 10. By some interface we must propagate the measurement all the way to GO > before libvirt starts the guest. > 11. GO verifies the measurement and if measurement matches then it may > give a secret blob -- which must be injected into the guest before > libvirt starts the VM. If verification failed, GO will request cloud > provider to destroy the VM. > 12. After secret blob is injected into guest, we call LAUNCH_FINISH > to destory the encryption context. > 13. libvirt issues "continue" command to resume the guest boot. > > Please note that the measurement value is protected with transport > encryption key (TIK) and it changes on each run. Similarly the secret blob > provided by GO does not need to be protected using libvirt/qemu APIs. The > secret is protected by TIK. From qemu and libvirt point of view these are > blobs and must be passed as-is to the SEV FW. So here's an alternative idea for starting guests: How about building a minimal shim firmware that runs on a single CPU and uses no hardware at all, it just contains the secret blob. That firmware just immediately stops and signals hypervisor that it is ready to be run in the cloud. Have user generate and start this shim firmware as a guest in a private setup, then export it out using SEND_* commands. Then instead of asking to launch guest, you ask provider to load it with RECEIVE_* commands. Unlike bios the shim firmware can hopefully be static so supporting it across qemu versions should be easy. The shim firmware then loads bios from qemu, verifies it in any way it sees fit (e.g. it could check a signature, version, etc: it is not limited to a hardware hash anymore). It then jumps to the bios. While not exactly the same, there is some similarity here with how people solved the issues around secureboot - by using a shim.

On Wed, Sep 27, 2017 at 08:39:24AM -0500, Brijesh Singh wrote: > Hi Michael, > > > On 09/26/2017 09:36 AM, Michael S. Tsirkin wrote: > > ... > >>> 8. libvirt launches the guest with "-S" >>> 9. While creating the SEV guest qemu does the following >>> i) create encryption context using GO's DH, session-info and guest policy >>> (LAUNCH_START) >>> ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) >>> iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement >> >> This part troubles me. This seems to mean that the guest being launched >> must know what the measurement of the bios is going to be. This means >> that the cloud provider can not update the bios without breaking guests. >> Also, while in practice you typically can run an old bios image on a new >> qemu instance, this is not really tested so would be very hard to >> support properly in QEMU. >> >> > > > The guest itself does not need to know the measurement of the bios -- > the SEV launch flow empowers the GO to validate the bootstrap code (bios) > before GO can provide a confidential information to the guest. Please note > that the validating of the measurement flow is optional. GO can ask cloud > provider to ignore the measurement all together and boot the SEV guest. As the OS runs on top of the bios, it does not seem prudent to boot the SEV guest without a way to verify that the bios is safe to use. Linux generally trusts the firmware it runs on. Are you looking for examples of how a malicious firmware can leak info out of the guest? > The measurement flow can be useful when GO decides to provide a custom > bios and want to know that his bios is used for booting the guest. In this > case, since the guest owner knows the initial contents of the guest at boot, > he can request the measurement from the cloud provider and compare it with > what the guest owner expects. I agree the measurement works for this, but If that's the only case, I'd say it's not all that interesting since most people will use a standard bios. I do think something needs to vaidate the bios though, and I do not think naively measuring the hash of the full bios is a way to do this that we can support well long term. > > >> And this looks like a fundamental problem with the hash based >> measurement that's in hardware. So below I suggest that we layer >> some software on top to rely on the hash as little as possible. >> >> >> >>> 10. By some interface we must propagate the measurement all the way to GO >>> before libvirt starts the guest. >>> 11. GO verifies the measurement and if measurement matches then it may >>> give a secret blob -- which must be injected into the guest before >>> libvirt starts the VM. If verification failed, GO will request cloud >>> provider to destroy the VM. >>> 12. After secret blob is injected into guest, we call LAUNCH_FINISH >>> to destory the encryption context. >>> 13. libvirt issues "continue" command to resume the guest boot. >>> >>> Please note that the measurement value is protected with transport >>> encryption key (TIK) and it changes on each run. Similarly the secret blob >>> provided by GO does not need to be protected using libvirt/qemu APIs. The >>> secret is protected by TIK. From qemu and libvirt point of view these are >>> blobs and must be passed as-is to the SEV FW. >> >> So here's an alternative idea for starting guests: >> >> How about building a minimal shim firmware that >> runs on a single CPU and uses no hardware at all, >> it just contains the secret blob. >> >> That firmware just immediately stops and signals >> hypervisor that it is ready to be run in the cloud. >> >> Have user generate and start this shim firmware as a guest in a private >> setup, then export it out using SEND_* commands. >> >> Then instead of asking to launch guest, you ask provider >> to load it with RECEIVE_* commands. >> >> Unlike bios the shim firmware >> can hopefully be static so supporting it across qemu >> versions should be easy. >> >> The shim firmware then loads bios from qemu, verifies >> it in any way it sees fit (e.g. it could check a signature, version, etc: >> it is not limited to a hardware hash anymore). >> It then jumps to the bios. >> >> >> While not exactly the same, there is some similarity >> here with how people solved the issues around secureboot - >> by using a shim. >>

On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote: > Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, > or even a BIOS+kernel+initrd is really not too significant. What is > significant is that the GO has a basis for trusting all code that is > imported in to their VM by the CP. And that NONE of the code provided by the > CP is "unknown" and unauditable by the GO. If the CP has a way to inject > code unknown to the GO in to the guest VM, the trust model is broken and > both GO and CP suffer the consequences. Absolutely. > When the CP needs to update the BIOS image, they will have to inform the GO > and allow the GO to establish trust in the CP's new BIOS image somehow. This GO update on every BIOS change is imho is not a workable model. You want something like checking the BIOS signature instead. And since hardware is all hash based, you need the shim to do it in software.

On 9/29/17 2:48 PM, Richard Relph wrote: > On 9/29/17 2:34 PM, Michael S. Tsirkin wrote: > > On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote: > > > Whether the "BIOS" is a "static shim" as Michael suggests, or a > > > full BIOS, > > > or even a BIOS+kernel+initrd is really not too significant. What is > > > significant is that the GO has a basis for trusting all code that is > > > imported in to their VM by the CP. And that NONE of the code > > > provided by the > > > CP is "unknown" and unauditable by the GO. If the CP has a way to inject > > > code unknown to the GO in to the guest VM, the trust model is broken and > > > both GO and CP suffer the consequences. > > > > Absolutely. > > > > > When the CP needs to update the BIOS image, they will have to > > > inform the GO > > > and allow the GO to establish trust in the CP's new BIOS image somehow. > > > > This GO update on every BIOS change is imho is not a workable model. You > > want something like checking the BIOS signature instead. And since > > hardware is all hash based, you need the shim to do it in software. > > A BIOS "signed" by the CP doesn't meet the security requirement. It is > code that is "unknown" to the GO. > > The (legitimate) CP does NOT want to be in that position of trust. If > they are, then some government somewhere is going to insist that they > sign a BIOS that allows the government to spy on the GO's VMs, and steal > secrets from it. Or some hacker admin will do it "for fun". > > How often do large public CPs really change their BIOSes? My sense is > that large public CPs prefer stability over "latest and greatest". > > And, perhaps more importantly, if a CP are able to sell a "more secure" > VM, one that justifies a higher price per vCPU hour, wouldn't that > warrant some changes in the "insecure" model being used today? Ultimately, I think both approaches are "doable". It will be a CP and GO decision. If the GO trusts the CP, the shim+signed BIOS will work fine.

If GO requires a more secure VM and the CP wants to offer it, the CP will figure out a way to satisfy the GO's "trust issue" that the BIOS can't be used to circumvent SEV's protections. Depending on your level of paranoia, that may require advance notice of BIOS changes, or even allowing the GO to provide the BIOS themselves, written to a spec supported by the CP's HV, and/or based on BIOS code provided by the CP.

It's a business decision and I think SEV can support both. That said, AMD currently has no plans to write a shim that can verify the signature on a CP-provided BIOS image. Richard

> > Richard

It's a business decision and I think SEV can support both.

On Fri, Sep 29, 2017 at 02:48:45PM -0500, Richard Relph wrote: > On 9/29/17 2:34 PM, Michael S. Tsirkin wrote: >> On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote: >>> Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, >>> or even a BIOS+kernel+initrd is really not too significant. What is >>> significant is that the GO has a basis for trusting all code that is >>> imported in to their VM by the CP. And that NONE of the code provided by the >>> CP is "unknown" and unauditable by the GO. If the CP has a way to inject >>> code unknown to the GO in to the guest VM, the trust model is broken and >>> both GO and CP suffer the consequences. >> >> Absolutely. >> >>> When the CP needs to update the BIOS image, they will have to inform the GO >>> and allow the GO to establish trust in the CP's new BIOS image somehow. >> >> This GO update on every BIOS change is imho is not a workable model. You >> want something like checking the BIOS signature instead. And since >> hardware is all hash based, you need the shim to do it in software. > > A BIOS "signed" by the CP doesn't meet the security requirement. It is code > that is "unknown" to the GO. There is a misunderstanding here. BIOS would not be signed by a CP. It would be signed by a trusted software vendor e.g. by Red Hat. > The (legitimate) CP does NOT want to be in that position of trust. If they > are, then some government somewhere is going to insist that they sign a BIOS > that allows the government to spy on the GO's VMs, and steal secrets from > it. Or some hacker admin will do it "for fun". > > How often do large public CPs really change their BIOSes? My sense is that > large public CPs prefer stability over "latest and greatest". CPs just do dnf update. Software vendors change BIOSes. And we do change them. Look at number of revisions for seabios in e.g. Fedora. More importantly we might need to change them quickly e.g. because of a security issue. Adding the need to coordinate with all GOs is not going to work. Neither can QEMU support booting old BIOS versions on new machine types indefinitely. > And, perhaps more importantly, if a CP are able to sell a "more secure" VM, > one that justifies a higher price per vCPU hour, wouldn't that warrant some > changes in the "insecure" model being used today? > > Richard Absolutely. CPs have no business signing images. But it is just not really feasible for software vendors to distribute hashes.

On 9/29/17 2:34 PM, Michael S. Tsirkin wrote: > On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote: > > Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, > > or even a BIOS+kernel+initrd is really not too significant. What is > > significant is that the GO has a basis for trusting all code that is > > imported in to their VM by the CP. And that NONE of the code provided by the > > CP is "unknown" and unauditable by the GO. If the CP has a way to inject > > code unknown to the GO in to the guest VM, the trust model is broken and > > both GO and CP suffer the consequences. > > Absolutely. > > > When the CP needs to update the BIOS image, they will have to inform the GO > > and allow the GO to establish trust in the CP's new BIOS image somehow. > > This GO update on every BIOS change is imho is not a workable model. You > want something like checking the BIOS signature instead. And since > hardware is all hash based, you need the shim to do it in software. A BIOS "signed" by the CP doesn't meet the security requirement. It is code that is "unknown" to the GO. The (legitimate) CP does NOT want to be in that position of trust. If they are, then some government somewhere is going to insist that they sign a BIOS that allows the government to spy on the GO's VMs, and steal secrets from it. Or some hacker admin will do it "for fun". How often do large public CPs really change their BIOSes? My sense is that large public CPs prefer stability over "latest and greatest".

And, perhaps more importantly, if a CP are able to sell a "more secure" VM, one that justifies a higher price per vCPU hour, wouldn't that warrant some changes in the "insecure" model being used today?

The expansion ROMs (containing UEFI drivers) of emulated PCI devices, and the same of assigned physical PCI devices, constitute another channel through which code enters the guest from the outside (i.e., from the Cloud Provider). The ROM BARs from which the guest firmware reads the UEFI binaries are not guest RAM, they are MMIO. (For execution, the drivers are copied into encrypted guest RAM.) If the guest has Secure Boot enabled, then the oproms are verified[*] (and not launched if verification fails), but this is slightly different from what I understand under audit-by-GO. It means the GO wouldn't get a measurement of the oproms for one-by-one clearing, when about to green-light a guest startup. Instead the GO would ensure that Secure Boot be enabled with the right certificates (and/or executable hashes) enrolled off the bat, and then implicitly trust all oprom drivers accepted by those certs / hashes. It's another layer of indirection. This is likely nothing new qualitatively, but "the devil is in the details", so I thought it was worth raising. [*] For edk2 / OvmfPkg specifics, I'll mention gEfiSecurityPkgTokenSpaceGuid.PcdOptionRomImageVerificationPolicy The SecurityPkg default is 0x04 ("Deny execution when there is security violation"). However, OVMF sets it to 0x00 ("Always trust the image"). Please see the following commit for the reasons: https://github.com/tianocore/edk2/commit/1fea9ddb4e3fd Brijesh, for SEV guests, we likely want to flip this PCD to 0x04, in the AmdSevInitialize() function, in "OvmfPkg/PlatformPei/AmdSev.c". For that we'll also have to change the PCD from fixed-at-build to dynamic, but that in turn will require a change to "SecurityPkg.dec" itself (currently it only allows fixed-at-build or patchable, not dynamic). Do you want me to file a BZ in the TianoCore tracker for this, and assign it to you? If you don't have time for writing the patch, I'm glad to do it too, but then the review could be slower; both other OvmfPkg co-maintainers are busy with other things.)

(3) Implement SEV encryption for pflash. A pflash chip can be in one of two modes: (a) it reads and executes as ROM, or (b) it behaves like a programmable (r/w) device with MMIO registers. Switching between both modes is explicit (see "OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlash.c"); perhaps some SEV controls could be hooked in there.