[libvirt] libvirt/QEMU/SEV interaction
Hi All, (sorry for the long message) CPUs from AMD EPYC family supports Secure Encrypted Virtualization (SEV) feature - the feature allows running encrypted VMs. To enable the feature, I have been submitting patches to Linux kernel [1], Qemu [2] and OVMF [3]. We have been making some good progress in getting patches accepted upstream in Linux and OVMF trees. SEV builds upon SME (Secure Memory Encryption) feature -- SME support just got pulled into 4.14 merge window. The base SEV patches are accepted in OVMF tree -- now we have SEV aware guest BIOS. I am getting ready to take off "RFC" tag from remaining patches to get them reviewed and accepted. The boot flow for launching an SEV guest is a bit different from a typical guest launch. In order to launch SEV guest from virt-manager or other high-level VM management tools, we need to design and implement new interface between libvirt and qemu, and probably add new APIs in libvirt to be used by VM management tools. I am new to the libvirt and need some expert advice while designing this interface. A pictorial representation for a SEV guest launch flow is available in SEV Spec Appendix A [4]. A typical flow looks like this: 1. Guest owner (GO) asks the cloud provider to launch SEV guest. 2. VM tool asks libvirt to provide its Platform Diffie-Hellman (PDH) key. 3. libvirt opens /dev/sev device to get its PDH and return the blob to the caller. 4. VM tool gives its PDH to GO. 5. GO provides its DH key, session-info and guest policy. 6. VM tool somehow communicates the GO provided information to libvirt. 7. libvirt adds "sev-guest" object in its xml file with all the information obtained from #5 (currently my xml file looks like this) <qemu:arg value='-object'> <qemu:arg value='sev-guest,id=sev0,policy=<GO_policy>,dh-key-file=<filename>,session-file=<filename>/> <qemu:arg value='-machine'/> <qemu:arg value='memory-encryption=sev0'/> 8. libvirt launches the guest with "-S" 9. While creating the SEV guest qemu does the following i) create encryption context using GO's DH, session-info and guest policy (LAUNCH_START) ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement 10. By some interface we must propagate the measurement all the way to GO before libvirt starts the guest. 11. GO verifies the measurement and if measurement matches then it may give a secret blob -- which must be injected into the guest before libvirt starts the VM. If verification failed, GO will request cloud provider to destroy the VM. 12. After secret blob is injected into guest, we call LAUNCH_FINISH to destory the encryption context. 13. libvirt issues "continue" command to resume the guest boot. Please note that the measurement value is protected with transport encryption key (TIK) and it changes on each run. Similarly the secret blob provided by GO does not need to be protected using libvirt/qemu APIs. The secret is protected by TIK. From qemu and libvirt point of view these are blobs and must be passed as-is to the SEV FW. Questions: a) Do we need to add a new set of APIs in libvirt to return the PDH from libvirt and VM tool ? Or can we use some pre-existing APIs to pass the opaque blobs ? (this is mainly for step 3 and 6) b) do we need to define a new xml tag to for memory-encryption ? or just use the qemu:args tag ? (step 6) c) what existing communicate interface can be used between libvirt and qemu to get the measurement ? can we add a new qemu monitor command 'get_sev_measurement' to get the measurement ? (step 10) d) how to pass the secret blob from libvirt to qemu ? should we consider adding a new object (sev-guest-secret) -- libvirt can add the object through qemu monitor. [1] https://marc.info/?l=kvm&m=150092661105069&w=2 [2] https://marc.info/?l=qemu-devel&m=148901186615642&w=2 [3] https://lists.01.org/pipermail/edk2-devel/2017-July/012220.html [4] http://support.amd.com/TechDocs/55766_SEV-KM%20API_Specification.pdf Thanks Brijesh
On Fri, Sep 08, 2017 at 06:57:30AM -0500, Brijesh Singh wrote:
Hi All,
(sorry for the long message)
CPUs from AMD EPYC family supports Secure Encrypted Virtualization (SEV) feature - the feature allows running encrypted VMs. To enable the feature, I have been submitting patches to Linux kernel [1], Qemu [2] and OVMF [3]. We have been making some good progress in getting patches accepted upstream in Linux and OVMF trees. SEV builds upon SME (Secure Memory Encryption) feature -- SME support just got pulled into 4.14 merge window. The base SEV patches are accepted in OVMF tree -- now we have SEV aware guest BIOS. I am getting ready to take off "RFC" tag from remaining patches to get them reviewed and accepted.
The boot flow for launching an SEV guest is a bit different from a typical guest launch. In order to launch SEV guest from virt-manager or other high-level VM management tools, we need to design and implement new interface between libvirt and qemu, and probably add new APIs in libvirt to be used by VM management tools. I am new to the libvirt and need some expert advice while designing this interface. A pictorial representation for a SEV guest launch flow is available in SEV Spec Appendix A [4].
A typical flow looks like this:
1. Guest owner (GO) asks the cloud provider to launch SEV guest. 2. VM tool asks libvirt to provide its Platform Diffie-Hellman (PDH) key. 3. libvirt opens /dev/sev device to get its PDH and return the blob to the caller.
What sort of size are we talking about for the PDH ? There's a few ways libvirt could report it 1. As an XML element in the host capabilities XML 2. As an XML element in the emulator capabilities XML 3. Via a newly added host API
4. VM tool gives its PDH to GO. 5. GO provides its DH key, session-info and guest policy.
Are steps 4 & 5 strictly required to be in this order, or is it possible What are the security requirements around the DH key, session info and guest policy ? Are any of them sensitive data which needs to be kept private from untrustworthy users. Also are all three of these items different for every guest launched, or are some of them likely to be the same for every guest ? eg, would the same guest policy blob be used for every guest, with only session info changing ? Also what sort of size are we talking about for each of these data items, KBs, 10's of KB, 100's of KBs or larger ? The security and data size can influence our design approach from the libvirt POV.
6. VM tool somehow communicates the GO provided information to libvirt.
Essentially we have two choices 1. inline in the guest XML config description passed to libvirt when defining the guest XML 2. out of band, ahead of time, via some other API prior to defining the guest XML, which is then referenced in guest XML. THis could be done using the virSecret APIs, where we create 3 secrets, one each for DH key, session-info and guets policy. The UUID of the secret could be specified in the guest XML. This has flexibility of allowing the same secrets ot be used for many guests (if this is valid for SEV)
7. libvirt adds "sev-guest" object in its xml file with all the information obtained from #5
(currently my xml file looks like this)
<qemu:arg value='-object'> <qemu:arg value='sev-guest,id=sev0,policy=<GO_policy>,dh-key-file=<filename>,session-file=<filename>/> <qemu:arg value='-machine'/> <qemu:arg value='memory-encryption=sev0'/>
8. libvirt launches the guest with "-S"
All libvirt guests get launched with -S, to give libvirt chance to do some setup before starting vCPUs. Normally vCPUs are started by default, but the VIR_DOMAIN_START_PAUSED flag allows the mgmt app to tell libvirt to leave vCPUS paused. Alternatively, if libvirt sees presencese of 'sev' config for the guest, it could automatically leave it in PAUSED state regardless of the VIR_DOMAIN_START_PAUSED flag.
9. While creating the SEV guest qemu does the following i) create encryption context using GO's DH, session-info and guest policy (LAUNCH_START) ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement 10. By some interface we must propagate the measurement all the way to GO before libvirt starts the guest.
Again, what kind of size data are we talking about for athe "measurement" blob ? a KB, 10's of KB, or more ? My first gut instinct would be for QEMU to emit a QMP event when it has the measurement available. The event could include the actual data blob, or we can could add an explicit QMP command to fetch the data blob. Libvirt could listen for this QEMU event, and in turn emit an event from libvirt with the same data, which the mgmt tool can finally give to the GO.
11. GO verifies the measurement and if measurement matches then it may give a secret blob -- which must be injected into the guest before libvirt starts the VM. If verification failed, GO will request cloud provider to destroy the VM.
So we need some mechanism to provide the secret blob. This could be done via a new libvirt API. Alternatively, if we're using virSecret for the other stuff, the guest config XML could include the UUID of a 4th secret. Libvirt would then watch to see when that secret has a value set, and pass that onto QEMU.
12. After secret blob is injected into guest, we call LAUNCH_FINISH to destory the encryption context. 13. libvirt issues "continue" command to resume the guest boot.
This is as simple as the mgmt tool calling virDomainResume to unpause CPUs. We could have it such that virDomainResume checks whether the virSecret has been populated secret blob by GO, and pass it onto QEMU at this time.
Please note that the measurement value is protected with transport encryption key (TIK) and it changes on each run. Similarly the secret blob provided by GO does not need to be protected using libvirt/qemu APIs. The secret is protected by TIK. From qemu and libvirt point of view these are blobs and must be passed as-is to the SEV FW.
Questions: a) Do we need to add a new set of APIs in libvirt to return the PDH from libvirt and VM tool ? Or can we use some pre-existing APIs to pass the opaque blobs ? (this is mainly for step 3 and 6) b) do we need to define a new xml tag to for memory-encryption ? or just use the qemu:args tag ? (step 6)
<qemu:args> is explicitly only ever for ad-hoc testing. For anything that is to be used in production deployment we must explicitly model it in the XML. So we definitely need new XML defined.
c) what existing communicate interface can be used between libvirt and qemu to get the measurement ? can we add a new qemu monitor command 'get_sev_measurement' to get the measurement ? (step 10)
Yes, QMP commands seeem most likely.
d) how to pass the secret blob from libvirt to qemu ? should we consider adding a new object (sev-guest-secret) -- libvirt can add the object through qemu monitor.
Yeah, that looks like a viable option too.
[1] https://marc.info/?l=kvm&m=150092661105069&w=2 [2] https://marc.info/?l=qemu-devel&m=148901186615642&w=2 [3] https://lists.01.org/pipermail/edk2-devel/2017-July/012220.html [4] http://support.amd.com/TechDocs/55766_SEV-KM%20API_Specification.pdf
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
A few answers in line… On 9/8/17, 8:16 AM, "Daniel P. Berrange" <berrange@redhat.com> wrote: On Fri, Sep 08, 2017 at 06:57:30AM -0500, Brijesh Singh wrote: > Hi All, > > (sorry for the long message) > > CPUs from AMD EPYC family supports Secure Encrypted Virtualization (SEV) > feature - the feature allows running encrypted VMs. To enable the feature, > I have been submitting patches to Linux kernel [1], Qemu [2] and OVMF [3]. > We have been making some good progress in getting patches accepted upstream > in Linux and OVMF trees. SEV builds upon SME (Secure Memory Encryption) > feature -- SME support just got pulled into 4.14 merge window. The base > SEV patches are accepted in OVMF tree -- now we have SEV aware guest BIOS. > I am getting ready to take off "RFC" tag from remaining patches to get them > reviewed and accepted. > > The boot flow for launching an SEV guest is a bit different from a typical > guest launch. In order to launch SEV guest from virt-manager or other > high-level VM management tools, we need to design and implement new > interface between libvirt and qemu, and probably add new APIs in libvirt > to be used by VM management tools. I am new to the libvirt and need some > expert advice while designing this interface. A pictorial representation > for a SEV guest launch flow is available in SEV Spec Appendix A [4]. > > A typical flow looks like this: > > 1. Guest owner (GO) asks the cloud provider to launch SEV guest. > 2. VM tool asks libvirt to provide its Platform Diffie-Hellman (PDH) key. > 3. libvirt opens /dev/sev device to get its PDH and return the blob to the > caller. What sort of size are we talking about for the PDH ? The PDH blob is described in reference 4. It’s 0x824 bytes long… a bit over 2K bytes. PDH is “Platform Diffie-Hellman” key, public portion. There's a few ways libvirt could report it 1. As an XML element in the host capabilities XML 2. As an XML element in the emulator capabilities XML 3. Via a newly added host API > 4. VM tool gives its PDH to GO. > 5. GO provides its DH key, session-info and guest policy. Are steps 4 & 5 strictly required to be in this order, or is it possible Steps 4 and 5 must occur in that order. The data sent by the GO in the “session info” is encrypted and integrity protected with keys that the GO derives from the GO private Diffie-Hellman key and the PDH public Diffie-Hellman key. What are the security requirements around the DH key, session info and guest policy ? Are any of them sensitive data which needs to be kept private from untrustworthy users. Also are all three of these items different for every guest launched, or are some of them likely to be the same for every guest ? The PDH is not sensitive. It is relatively static for the platform. (It only changes when the platform owner chooses to change the apparent identity of the platform that will actually run the virtual machine.) The 128-byte session info data is encrypted and integrity protected by the GO. It need not be additionally encrypted or integrity protected. It should vary for EVERY guest launch. The 4-byte guest policy must be sent in the clear as some components may want to observe the policy bits. It may change from guest to guest, but there will likely only be a few common values. eg, would the same guest policy blob be used for every guest, with only session info changing ? Also what sort of size are we talking about for each of these data items, KBs, 10's of KB, 100's of KBs or larger ? The security and data size can influence our design approach from the libvirt POV. > 6. VM tool somehow communicates the GO provided information to libvirt. Essentially we have two choices 1. inline in the guest XML config description passed to libvirt when defining the guest XML 2. out of band, ahead of time, via some other API prior to defining the guest XML, which is then referenced in guest XML. THis could be done using the virSecret APIs, where we create 3 secrets, one each for DH key, session-info and guets policy. The UUID of the secret could be specified in the guest XML. This has flexibility of allowing the same secrets ot be used for many guests (if this is valid for SEV) > 7. libvirt adds "sev-guest" object in its xml file with all the information > obtained from #5 > > (currently my xml file looks like this) > > <qemu:arg value='-object'> > <qemu:arg > value='sev-guest,id=sev0,policy=<GO_policy>,dh-key-file=<filename>,session-file=<filename>/> > <qemu:arg value='-machine'/> > <qemu:arg value='memory-encryption=sev0'/> > > 8. libvirt launches the guest with "-S" All libvirt guests get launched with -S, to give libvirt chance to do some setup before starting vCPUs. Normally vCPUs are started by default, but the VIR_DOMAIN_START_PAUSED flag allows the mgmt app to tell libvirt to leave vCPUS paused. Alternatively, if libvirt sees presencese of 'sev' config for the guest, it could automatically leave it in PAUSED state regardless of the VIR_DOMAIN_START_PAUSED flag. While using the LAUNCH_MEASURE and LAUNCH_SECRET operations is expected to be the common case, they are optional. I would recommend requiring the upstream software to explicitly set the VIR_DOMAIN_START_PAUSED flag, if only to minimize the dependencies… > 9. While creating the SEV guest qemu does the following > i) create encryption context using GO's DH, session-info and guest policy > (LAUNCH_START) > ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) > iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement > 10. By some interface we must propagate the measurement all the way to GO > before libvirt starts the guest. Again, what kind of size data are we talking about for athe "measurement" blob ? a KB, 10's of KB, or more ? The measurement is 48 bytes… My first gut instinct would be for QEMU to emit a QMP event when it has the measurement available. The event could include the actual data blob, or we can could add an explicit QMP command to fetch the data blob. Libvirt could listen for this QEMU event, and in turn emit an event from libvirt with the same data, which the mgmt tool can finally give to the GO. > 11. GO verifies the measurement and if measurement matches then it may > give a secret blob -- which must be injected into the guest before > libvirt starts the VM. If verification failed, GO will request cloud > provider to destroy the VM. So we need some mechanism to provide the secret blob. This could be done via a new libvirt API. Alternatively, if we're using virSecret for the other stuff, the guest config XML could include the UUID of a 4th secret. Libvirt would then watch to see when that secret has a value set, and pass that onto QEMU. The secret is optional… it is up to 16KB, already encrypted, and integrity protected with an IV and MAC value passed with the secret. The guest address the secret should be deposited at is Technically, the SEV FW API allows there to be more than one secret… I don’t see a strong reason to require libvirt to support more than one, though. > 12. After secret blob is injected into guest, we call LAUNCH_FINISH > to destory the encryption context. > 13. libvirt issues "continue" command to resume the guest boot. This is as simple as the mgmt tool calling virDomainResume to unpause CPUs. We could have it such that virDomainResume checks whether the virSecret has been populated secret blob by GO, and pass it onto QEMU at this time. > Please note that the measurement value is protected with transport > encryption key (TIK) and it changes on each run. Similarly the secret blob > provided by GO does not need to be protected using libvirt/qemu APIs. The > secret is protected by TIK. From qemu and libvirt point of view these are > blobs and must be passed as-is to the SEV FW. > > Questions: > a) Do we need to add a new set of APIs in libvirt to return the PDH from > libvirt and VM tool ? Or can we use some pre-existing APIs to pass the > opaque blobs ? (this is mainly for step 3 and 6) > b) do we need to define a new xml tag to for memory-encryption ? or just > use the qemu:args tag ? (step 6) <qemu:args> is explicitly only ever for ad-hoc testing. For anything that is to be used in production deployment we must explicitly model it in the XML. So we definitely need new XML defined. > c) what existing communicate interface can be used between libvirt and qemu > to get the measurement ? can we add a new qemu monitor command > 'get_sev_measurement' to get the measurement ? (step 10) Yes, QMP commands seeem most likely. > d) how to pass the secret blob from libvirt to qemu ? should we consider > adding a new object (sev-guest-secret) -- libvirt can add the object through > qemu monitor. Yeah, that looks like a viable option too. > > > [1] https://marc.info/?l=kvm&m=150092661105069&w=2 > [2] https://marc.info/?l=qemu-devel&m=148901186615642&w=2 > [3] https://lists.01.org/pipermail/edk2-devel/2017-July/012220.html > [4] http://support.amd.com/TechDocs/55766_SEV-KM%20API_Specification.pdf Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Sep 08, 2017 at 01:45:06PM +0000, Relph, Richard wrote:
A few answers in line…
On 9/8/17, 8:16 AM, "Daniel P. Berrange" <berrange@redhat.com> wrote:
On Fri, Sep 08, 2017 at 06:57:30AM -0500, Brijesh Singh wrote: > Hi All, > > (sorry for the long message) > > CPUs from AMD EPYC family supports Secure Encrypted Virtualization (SEV) > feature - the feature allows running encrypted VMs. To enable the feature, > I have been submitting patches to Linux kernel [1], Qemu [2] and OVMF [3]. > We have been making some good progress in getting patches accepted upstream > in Linux and OVMF trees. SEV builds upon SME (Secure Memory Encryption) > feature -- SME support just got pulled into 4.14 merge window. The base > SEV patches are accepted in OVMF tree -- now we have SEV aware guest BIOS. > I am getting ready to take off "RFC" tag from remaining patches to get them > reviewed and accepted. > > The boot flow for launching an SEV guest is a bit different from a typical > guest launch. In order to launch SEV guest from virt-manager or other > high-level VM management tools, we need to design and implement new > interface between libvirt and qemu, and probably add new APIs in libvirt > to be used by VM management tools. I am new to the libvirt and need some > expert advice while designing this interface. A pictorial representation > for a SEV guest launch flow is available in SEV Spec Appendix A [4]. > > A typical flow looks like this: > > 1. Guest owner (GO) asks the cloud provider to launch SEV guest. > 2. VM tool asks libvirt to provide its Platform Diffie-Hellman (PDH) key. > 3. libvirt opens /dev/sev device to get its PDH and return the blob to the > caller.
What sort of size are we talking about for the PDH ?
The PDH blob is described in reference 4. It’s 0x824 bytes long… a bit over 2K bytes. PDH is “Platform Diffie-Hellman” key, public portion.
There's a few ways libvirt could report it
1. As an XML element in the host capabilities XML 2. As an XML element in the emulator capabilities XML 3. Via a newly added host API
> 4. VM tool gives its PDH to GO. > 5. GO provides its DH key, session-info and guest policy.
Are steps 4 & 5 strictly required to be in this order, or is it possible
Steps 4 and 5 must occur in that order. The data sent by the GO in the “session info” is encrypted and integrity protected with keys that the GO derives from the GO private Diffie-Hellman key and the PDH public Diffie-Hellman key.
What are the security requirements around the DH key, session info and guest policy ? Are any of them sensitive data which needs to be kept private from untrustworthy users. Also are all three of these items different for every guest launched, or are some of them likely to be the same for every guest ?
The PDH is not sensitive. It is relatively static for the platform. (It only changes when the platform owner chooses to change the apparent identity of the platform that will actually run the virtual machine.) The 128-byte session info data is encrypted and integrity protected by the GO. It need not be additionally encrypted or integrity protected. It should vary for EVERY guest launch. The 4-byte guest policy must be sent in the clear as some components may want to observe the policy bits. It may change from guest to guest, but there will likely only be a few common values.
Given this, and the pretty small data sizes, I think this info could in fact all be provided inline in the XML config - either hex or base64 encoded for the binary blobs.
> 8. libvirt launches the guest with "-S"
All libvirt guests get launched with -S, to give libvirt chance to do some setup before starting vCPUs. Normally vCPUs are started by default, but the VIR_DOMAIN_START_PAUSED flag allows the mgmt app to tell libvirt to leave vCPUS paused.
Alternatively, if libvirt sees presencese of 'sev' config for the guest, it could automatically leave it in PAUSED state regardless of the VIR_DOMAIN_START_PAUSED flag.
While using the LAUNCH_MEASURE and LAUNCH_SECRET operations is expected to be the common case, they are optional. I would recommend requiring the upstream software to explicitly set the VIR_DOMAIN_START_PAUSED flag, if only to minimize the dependencies…
Ok.
> 9. While creating the SEV guest qemu does the following > i) create encryption context using GO's DH, session-info and guest policy > (LAUNCH_START) > ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) > iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement > 10. By some interface we must propagate the measurement all the way to GO > before libvirt starts the guest.
Again, what kind of size data are we talking about for athe "measurement" blob ? a KB, 10's of KB, or more ?
The measurement is 48 bytes…
For that small size we can definitely provide is inline in the event payload then.
My first gut instinct would be for QEMU to emit a QMP event when it has the measurement available. The event could include the actual data blob, or we can could add an explicit QMP command to fetch the data blob.
Libvirt could listen for this QEMU event, and in turn emit an event from libvirt with the same data, which the mgmt tool can finally give to the GO.
> 11. GO verifies the measurement and if measurement matches then it may > give a secret blob -- which must be injected into the guest before > libvirt starts the VM. If verification failed, GO will request cloud > provider to destroy the VM.
So we need some mechanism to provide the secret blob. This could be done via a new libvirt API. Alternatively, if we're using virSecret for the other stuff, the guest config XML could include the UUID of a 4th secret. Libvirt would then watch to see when that secret has a value set, and pass that onto QEMU.
The secret is optional… it is up to 16KB, already encrypted, and integrity protected with an IV and MAC value passed with the secret. The guest address the secret should be deposited at is Technically, the SEV FW API allows there to be more than one secret… I don’t see a strong reason to require libvirt to support more than one, though.
So I think we could simply add a new API to libvirt to provide this data item.
> 12. After secret blob is injected into guest, we call LAUNCH_FINISH > to destory the encryption context. > 13. libvirt issues "continue" command to resume the guest boot.
This is as simple as the mgmt tool calling virDomainResume to unpause CPUs. We could have it such that virDomainResume checks whether the virSecret has been populated secret blob by GO, and pass it onto QEMU at this time.
> Please note that the measurement value is protected with transport > encryption key (TIK) and it changes on each run. Similarly the secret blob > provided by GO does not need to be protected using libvirt/qemu APIs. The > secret is protected by TIK. From qemu and libvirt point of view these are > blobs and must be passed as-is to the SEV FW. > > Questions: > a) Do we need to add a new set of APIs in libvirt to return the PDH from > libvirt and VM tool ? Or can we use some pre-existing APIs to pass the > opaque blobs ? (this is mainly for step 3 and 6) > b) do we need to define a new xml tag to for memory-encryption ? or just > use the qemu:args tag ? (step 6)
<qemu:args> is explicitly only ever for ad-hoc testing.
For anything that is to be used in production deployment we must explicitly model it in the XML. So we definitely need new XML defined.
> c) what existing communicate interface can be used between libvirt and qemu > to get the measurement ? can we add a new qemu monitor command > 'get_sev_measurement' to get the measurement ? (step 10)
Yes, QMP commands seeem most likely.
> d) how to pass the secret blob from libvirt to qemu ? should we consider > adding a new object (sev-guest-secret) -- libvirt can add the object through > qemu monitor.
Yeah, that looks like a viable option too.
So I could see a flow like the following: 1. mgmt tool calls virConnectGetCapabilities. This returns an XML document that includes the following <host> ...other bits... <sev> <platform-key>...hex encoded PDH key...</platform-key> </sev> </host> 2. mgmt tool requests to start a guest calling virCreateXML(), passing VIR_DOMAIN_START_PAUSED. The XML would include <sev> <owner-key>...hex encode DH key...</owner-key> <session-info>..hex encode info...</session-info> <policy>...int32 value..</policy> </sev> if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, libvirt would report an error and refuse to start the guest 3. Libvirt generates the QEMU cli arg to enable SEV using the XML data and starts QEMU, leaving CPUs paused 4. QEMU emits a SEV_MEASURE event containing the measurement blob 5. Libvirt catches the QEMU event and emits its own VIR_CONNECT_DOMAIN_EVENT_SEV_MEASURE event containing the measurement blob 6. GO does its validation of the measurement 7a If validation failed, then virDomainDestroy() to stop QEMU 7b If validation succeeed Optionally call virDomainSetSEVSecret() providing the optional secret, then virDomainResume() to let QEMU continue Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi Daniel, On 09/08/2017 09:52 AM, Daniel P. Berrange wrote:
On Fri, Sep 08, 2017 at 01:45:06PM +0000, Relph, Richard wrote:
A few answers in line…
On 9/8/17, 8:16 AM, "Daniel P. Berrange" <berrange@redhat.com> wrote:
On Fri, Sep 08, 2017 at 06:57:30AM -0500, Brijesh Singh wrote: > Hi All, > > (sorry for the long message) > > CPUs from AMD EPYC family supports Secure Encrypted Virtualization (SEV) > feature - the feature allows running encrypted VMs. To enable the feature, > I have been submitting patches to Linux kernel [1], Qemu [2] and OVMF [3]. > We have been making some good progress in getting patches accepted upstream > in Linux and OVMF trees. SEV builds upon SME (Secure Memory Encryption) > feature -- SME support just got pulled into 4.14 merge window. The base > SEV patches are accepted in OVMF tree -- now we have SEV aware guest BIOS. > I am getting ready to take off "RFC" tag from remaining patches to get them > reviewed and accepted. > > The boot flow for launching an SEV guest is a bit different from a typical > guest launch. In order to launch SEV guest from virt-manager or other > high-level VM management tools, we need to design and implement new > interface between libvirt and qemu, and probably add new APIs in libvirt > to be used by VM management tools. I am new to the libvirt and need some > expert advice while designing this interface. A pictorial representation > for a SEV guest launch flow is available in SEV Spec Appendix A [4]. > > A typical flow looks like this: > > 1. Guest owner (GO) asks the cloud provider to launch SEV guest. > 2. VM tool asks libvirt to provide its Platform Diffie-Hellman (PDH) key. > 3. libvirt opens /dev/sev device to get its PDH and return the blob to the > caller.
What sort of size are we talking about for the PDH ?
The PDH blob is described in reference 4. It’s 0x824 bytes long… a bit over 2K bytes. PDH is “Platform Diffie-Hellman” key, public portion.
There's a few ways libvirt could report it
1. As an XML element in the host capabilities XML 2. As an XML element in the emulator capabilities XML 3. Via a newly added host API
> 4. VM tool gives its PDH to GO. > 5. GO provides its DH key, session-info and guest policy.
Are steps 4 & 5 strictly required to be in this order, or is it possible
Steps 4 and 5 must occur in that order. The data sent by the GO in the “session info” is encrypted and integrity protected with keys that the GO derives from the GO private Diffie-Hellman key and the PDH public Diffie-Hellman key.
What are the security requirements around the DH key, session info and guest policy ? Are any of them sensitive data which needs to be kept private from untrustworthy users. Also are all three of these items different for every guest launched, or are some of them likely to be the same for every guest ?
The PDH is not sensitive. It is relatively static for the platform. (It only changes when the platform owner chooses to change the apparent identity of the platform that will actually run the virtual machine.) The 128-byte session info data is encrypted and integrity protected by the GO. It need not be additionally encrypted or integrity protected. It should vary for EVERY guest launch. The 4-byte guest policy must be sent in the clear as some components may want to observe the policy bits. It may change from guest to guest, but there will likely only be a few common values.
Given this, and the pretty small data sizes, I think this info could in fact all be provided inline in the XML config - either hex or base64 encoded for the binary blobs.
> 8. libvirt launches the guest with "-S"
All libvirt guests get launched with -S, to give libvirt chance to do some setup before starting vCPUs. Normally vCPUs are started by default, but the VIR_DOMAIN_START_PAUSED flag allows the mgmt app to tell libvirt to leave vCPUS paused.
Alternatively, if libvirt sees presencese of 'sev' config for the guest, it could automatically leave it in PAUSED state regardless of the VIR_DOMAIN_START_PAUSED flag.
While using the LAUNCH_MEASURE and LAUNCH_SECRET operations is expected to be the common case, they are optional. I would recommend requiring the upstream software to explicitly set the VIR_DOMAIN_START_PAUSED flag, if only to minimize the dependencies…
Ok.
> 9. While creating the SEV guest qemu does the following > i) create encryption context using GO's DH, session-info and guest policy > (LAUNCH_START) > ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) > iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement > 10. By some interface we must propagate the measurement all the way to GO > before libvirt starts the guest.
Again, what kind of size data are we talking about for athe "measurement" blob ? a KB, 10's of KB, or more ?
The measurement is 48 bytes…
For that small size we can definitely provide is inline in the event payload then.
My first gut instinct would be for QEMU to emit a QMP event when it has the measurement available. The event could include the actual data blob, or we can could add an explicit QMP command to fetch the data blob.
Libvirt could listen for this QEMU event, and in turn emit an event from libvirt with the same data, which the mgmt tool can finally give to the GO.
> 11. GO verifies the measurement and if measurement matches then it may > give a secret blob -- which must be injected into the guest before > libvirt starts the VM. If verification failed, GO will request cloud > provider to destroy the VM.
So we need some mechanism to provide the secret blob. This could be done via a new libvirt API. Alternatively, if we're using virSecret for the other stuff, the guest config XML could include the UUID of a 4th secret. Libvirt would then watch to see when that secret has a value set, and pass that onto QEMU.
The secret is optional… it is up to 16KB, already encrypted, and integrity protected with an IV and MAC value passed with the secret. The guest address the secret should be deposited at is Technically, the SEV FW API allows there to be more than one secret… I don’t see a strong reason to require libvirt to support more than one, though.
So I think we could simply add a new API to libvirt to provide this data item.
> 12. After secret blob is injected into guest, we call LAUNCH_FINISH > to destory the encryption context. > 13. libvirt issues "continue" command to resume the guest boot.
This is as simple as the mgmt tool calling virDomainResume to unpause CPUs. We could have it such that virDomainResume checks whether the virSecret has been populated secret blob by GO, and pass it onto QEMU at this time.
> Please note that the measurement value is protected with transport > encryption key (TIK) and it changes on each run. Similarly the secret blob > provided by GO does not need to be protected using libvirt/qemu APIs. The > secret is protected by TIK. From qemu and libvirt point of view these are > blobs and must be passed as-is to the SEV FW. > > Questions: > a) Do we need to add a new set of APIs in libvirt to return the PDH from > libvirt and VM tool ? Or can we use some pre-existing APIs to pass the > opaque blobs ? (this is mainly for step 3 and 6) > b) do we need to define a new xml tag to for memory-encryption ? or just > use the qemu:args tag ? (step 6)
<qemu:args> is explicitly only ever for ad-hoc testing.
For anything that is to be used in production deployment we must explicitly model it in the XML. So we definitely need new XML defined.
> c) what existing communicate interface can be used between libvirt and qemu > to get the measurement ? can we add a new qemu monitor command > 'get_sev_measurement' to get the measurement ? (step 10)
Yes, QMP commands seeem most likely.
> d) how to pass the secret blob from libvirt to qemu ? should we consider > adding a new object (sev-guest-secret) -- libvirt can add the object through > qemu monitor.
Yeah, that looks like a viable option too.
So I could see a flow like the following:
The flow looks good
1. mgmt tool calls virConnectGetCapabilities. This returns an XML document that includes the following
<host> ...other bits... <sev> <platform-key>...hex encoded PDH key...</platform-key> </sev> </host>
2. mgmt tool requests to start a guest calling virCreateXML(), passing VIR_DOMAIN_START_PAUSED. The XML would include
<sev> <owner-key>...hex encode DH key...</owner-key> <session-info>..hex encode info...</session-info> <policy>...int32 value..</policy> </sev>
if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, libvirt would report an error and refuse to start the guest
One thing which is not clear to me is, how do we know that we are asked to launch SEV guest? Are you thinking that <sev> tag in the XML will hint libvirt that GO has asked to launch a SEV guest?
3. Libvirt generates the QEMU cli arg to enable SEV using the XML data and starts QEMU, leaving CPUs paused
I am looking at [1] to get the feel for how do we model it in the XML. As you can see I am using ad-hoc <qemu:args> to create the sev-guest object. Currently, sev-guest object accepts the following properties: dh-cert-file: <file containing the GO DH key> session-info-file: <file contain the GO session info> policy: <int32 GO policy> I believe the new XML model will influence the property input type, Any recommendation on how do model this part ? thank you so much. [1] https://libvirt.org/formatdomain.html#elementsCPU
4. QEMU emits a SEV_MEASURE event containing the measurement blob
5. Libvirt catches the QEMU event and emits its own VIR_CONNECT_DOMAIN_EVENT_SEV_MEASURE event containing the measurement blob
6. GO does its validation of the measurement
7a If validation failed, then virDomainDestroy() to stop QEMU
7b If validation succeeed
Optionally call
virDomainSetSEVSecret()
providing the optional secret, then
virDomainResume()
to let QEMU continue
Regards, Daniel
On Fri, Sep 08, 2017 at 10:48:10AM -0500, Brijesh Singh wrote:
So I could see a flow like the following:
The flow looks good
1. mgmt tool calls virConnectGetCapabilities. This returns an XML document that includes the following
<host> ...other bits... <sev> <platform-key>...hex encoded PDH key...</platform-key> </sev> </host>
2. mgmt tool requests to start a guest calling virCreateXML(), passing VIR_DOMAIN_START_PAUSED. The XML would include
<sev> <owner-key>...hex encode DH key...</owner-key> <session-info>..hex encode info...</session-info> <policy>...int32 value..</policy> </sev>
if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, libvirt would report an error and refuse to start the guest
One thing which is not clear to me is, how do we know that we are asked to launch SEV guest? Are you thinking that <sev> tag in the XML will hint libvirt that GO has asked to launch a SEV guest?
Yes, the existance of the <sev> tag is the indicator that informs libvirt that SEV *must* be used for the guest.
3. Libvirt generates the QEMU cli arg to enable SEV using the XML data and starts QEMU, leaving CPUs paused
I am looking at [1] to get the feel for how do we model it in the XML. As you can see I am using ad-hoc <qemu:args> to create the sev-guest object. Currently, sev-guest object accepts the following properties:
dh-cert-file: <file containing the GO DH key> session-info-file: <file contain the GO session info> policy: <int32 GO policy>
I believe the new XML model will influence the property input type, Any recommendation on how do model this part ? thank you so much.
That looks ok to me - even if QEMU wants the data provided in files on disk, libvirt can just create the files on the fly from the data it has in the <sev> element in the XML file. Since they're only needed during startup, libvirt can then easily delete the files the moment QEMU has completed its startup.
[1] https://libvirt.org/formatdomain.html#elementsCPU
4. QEMU emits a SEV_MEASURE event containing the measurement blob
5. Libvirt catches the QEMU event and emits its own VIR_CONNECT_DOMAIN_EVENT_SEV_MEASURE event containing the measurement blob
6. GO does its validation of the measurement
7a If validation failed, then virDomainDestroy() to stop QEMU
7b If validation succeeed
Optionally call
virDomainSetSEVSecret()
providing the optional secret, then
virDomainResume()
to let QEMU continue
Regards, Daniel
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 09/08/2017 10:51 AM, Daniel P. Berrange wrote:
On Fri, Sep 08, 2017 at 10:48:10AM -0500, Brijesh Singh wrote:
So I could see a flow like the following:
The flow looks good
1. mgmt tool calls virConnectGetCapabilities. This returns an XML document that includes the following
<host> ...other bits... <sev> <platform-key>...hex encoded PDH key...</platform-key> </sev> </host>
2. mgmt tool requests to start a guest calling virCreateXML(), passing VIR_DOMAIN_START_PAUSED. The XML would include
<sev> <owner-key>...hex encode DH key...</owner-key> <session-info>..hex encode info...</session-info> <policy>...int32 value..</policy> </sev>
if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, libvirt would report an error and refuse to start the guest
One thing which is not clear to me is, how do we know that we are asked to launch SEV guest? Are you thinking that <sev> tag in the XML will hint libvirt that GO has asked to launch a SEV guest?
Yes, the existance of the <sev> tag is the indicator that informs libvirt that SEV *must* be used for the guest.
Thanks for confirming.
3. Libvirt generates the QEMU cli arg to enable SEV using the XML data and starts QEMU, leaving CPUs paused
I am looking at [1] to get the feel for how do we model it in the XML. As you can see I am using ad-hoc <qemu:args> to create the sev-guest object. Currently, sev-guest object accepts the following properties:
dh-cert-file: <file containing the GO DH key> session-info-file: <file contain the GO session info> policy: <int32 GO policy>
I believe the new XML model will influence the property input type, Any recommendation on how do model this part ? thank you so much.
That looks ok to me - even if QEMU wants the data provided in files on disk, libvirt can just create the files on the fly from the data it has in the <sev> element in the XML file. Since they're only needed during startup, libvirt can then easily delete the files the moment QEMU has completed its startup.
Perfect! works well with me.
On 09/08/17 17:51, Daniel P. Berrange wrote:
On Fri, Sep 08, 2017 at 10:48:10AM -0500, Brijesh Singh wrote:
I am looking at [1] to get the feel for how do we model it in the XML. As you can see I am using ad-hoc <qemu:args> to create the sev-guest object. Currently, sev-guest object accepts the following properties:
dh-cert-file: <file containing the GO DH key> session-info-file: <file contain the GO session info> policy: <int32 GO policy>
I believe the new XML model will influence the property input type, Any recommendation on how do model this part ? thank you so much.
That looks ok to me - even if QEMU wants the data provided in files on disk, libvirt can just create the files on the fly from the data it has in the <sev> element in the XML file. Since they're only needed during startup, libvirt can then easily delete the files the moment QEMU has completed its startup.
/dev/fd/N filenames could be used for poor man's fd passing, I think. (/dev/fd is a symlink to the /proc/self/fd directory) proc(5) has documentation on this. Thanks, Laszlo
On Fri, Sep 08, 2017 at 10:48:10AM -0500, Brijesh Singh wrote:
> 11. GO verifies the measurement and if measurement matches then it may > give a secret blob -- which must be injected into the guest before > libvirt starts the VM. If verification failed, GO will request cloud > provider to destroy the VM.
I realised I'm missing something here: how does GO limit the secret to the specific VM? For example, what prevents hypervisor from launching two VMs with the same GO's DH, getting measurement from 1st one but injecting the secret into the second one? Thanks, -- MST
* Michael S. Tsirkin (mst@redhat.com) wrote:
On Fri, Sep 08, 2017 at 10:48:10AM -0500, Brijesh Singh wrote:
> 11. GO verifies the measurement and if measurement matches then it may > give a secret blob -- which must be injected into the guest before > libvirt starts the VM. If verification failed, GO will request cloud > provider to destroy the VM.
I realised I'm missing something here: how does GO limit the secret to the specific VM? For example, what prevents hypervisor from launching two VMs with the same GO's DH, getting measurement from 1st one but injecting the secret into the second one?
Isn't that the 'trusted channel nonce currently associated with the guest' in the guest context? Dave
Thanks,
-- MST
-- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
On Wed, Oct 18, 2017 at 08:18:48PM +0100, Dr. David Alan Gilbert wrote:
* Michael S. Tsirkin (mst@redhat.com) wrote:
On Fri, Sep 08, 2017 at 10:48:10AM -0500, Brijesh Singh wrote:
> 11. GO verifies the measurement and if measurement matches then it may > give a secret blob -- which must be injected into the guest before > libvirt starts the VM. If verification failed, GO will request cloud > provider to destroy the VM.
I realised I'm missing something here: how does GO limit the secret to the specific VM? For example, what prevents hypervisor from launching two VMs with the same GO's DH, getting measurement from 1st one but injecting the secret into the second one?
Isn't that the 'trusted channel nonce currently associated with the guest' in the guest context?
Dave
Let me try to clarify the question. I understand that sometimes a key is shared between VMs. If this is allowed, it seems that a hypervisor can run any number of VMs with the same key. An unauthorised VM will not get a measurement that guest owner authorizes, but can the hypervisor get secret intended for an authorized VM and then inject it into an unauthorized one sharing the same key?
Thanks,
-- MST
-- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
On 10/18/17 8:35 PM, Michael S. Tsirkin wrote:
On Wed, Oct 18, 2017 at 08:18:48PM +0100, Dr. David Alan Gilbert wrote:
* Michael S. Tsirkin (mst@redhat.com) wrote:
On Fri, Sep 08, 2017 at 10:48:10AM -0500, Brijesh Singh wrote:
> 11. GO verifies the measurement and if measurement matches then it may > give a secret blob -- which must be injected into the guest before > libvirt starts the VM. If verification failed, GO will request cloud > provider to destroy the VM.
I realised I'm missing something here: how does GO limit the secret to the specific VM? For example, what prevents hypervisor from launching two VMs with the same GO's DH, getting measurement from 1st one but injecting the secret into the second one?
Isn't that the 'trusted channel nonce currently associated with the guest' in the guest context?
Dave
Let me try to clarify the question. I understand that sometimes a key is shared between VMs. If this is allowed, it seems that a hypervisor can run any number of VMs with the same key. An unauthorised VM will not get a measurement that guest owner authorizes, but can the hypervisor get secret intended for an authorized VM and then inject it into an unauthorized one sharing the same key?
Michael, Yes, that's possible. This is why we recommend against a guest owner authorizing key sharing. There's no way for the guest owner to control what other guests the HV might install using the same key and mapping the same memory to. So a security-conscious customer, especially in a cloud environment, should never enable key sharing. Richard
Thanks,
-- MST
-- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
[...]
> c) what existing communicate interface can be used between libvirt and qemu > to get the measurement ? can we add a new qemu monitor command > 'get_sev_measurement' to get the measurement ? (step 10)
Yes, QMP commands seeem most likely.
> d) how to pass the secret blob from libvirt to qemu ? should we consider > adding a new object (sev-guest-secret) -- libvirt can add the object through > qemu monitor.
Yeah, that looks like a viable option too.
So I could see a flow like the following:
1. mgmt tool calls virConnectGetCapabilities. This returns an XML document that includes the following
<host> ...other bits... <sev> <platform-key>...hex encoded PDH key...</platform-key> </sev> </host>
2. mgmt tool requests to start a guest calling virCreateXML(), passing VIR_DOMAIN_START_PAUSED. The XML would include
<sev> <owner-key>...hex encode DH key...</owner-key> <session-info>..hex encode info...</session-info> <policy>...int32 value..</policy> </sev>
if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, libvirt would report an error and refuse to start the guest
3. Libvirt generates the QEMU cli arg to enable SEV using the XML data and starts QEMU, leaving CPUs paused
4. QEMU emits a SEV_MEASURE event containing the measurement blob
Speaking of which, I expect QEMU to have a QMP command to retrieve the measurement, in which case I think libvirt has to provide an API for the user to retrieve the measurement in case libvirtd crashes somewhere between setting up QEMU and waiting for the measurement event from QEMU, or simply because the GO missed the event for some unspecified reason.
5. Libvirt catches the QEMU event and emits its own VIR_CONNECT_DOMAIN_EVENT_SEV_MEASURE event containing the measurement blob
6. GO does its validation of the measurement
7a If validation failed, then virDomainDestroy() to stop QEMU
7b If validation succeeed
Optionally call
virDomainSetSEVSecret()
Given the fact that we're likely introducing a new <sev> element to the XML config, I'm more inclined to utilizing the existing virSecret interfaces (as was originally suggested) instead of creating a vendor-specific API. You could have an optional secret sub-element within the <sev> element and libvirt would simply check if that secret has a value set, once the GO issues virDomainResume(). Any particular reason for having a specific API for this that I'm missing? Other than that I like the initial proposal for the design. Erik
On Mon, Sep 18, 2017 at 11:43:57AM +0200, Erik Skultety wrote:
[...]
> c) what existing communicate interface can be used between libvirt and qemu > to get the measurement ? can we add a new qemu monitor command > 'get_sev_measurement' to get the measurement ? (step 10)
Yes, QMP commands seeem most likely.
> d) how to pass the secret blob from libvirt to qemu ? should we consider > adding a new object (sev-guest-secret) -- libvirt can add the object through > qemu monitor.
Yeah, that looks like a viable option too.
So I could see a flow like the following:
1. mgmt tool calls virConnectGetCapabilities. This returns an XML document that includes the following
<host> ...other bits... <sev> <platform-key>...hex encoded PDH key...</platform-key> </sev> </host>
2. mgmt tool requests to start a guest calling virCreateXML(), passing VIR_DOMAIN_START_PAUSED. The XML would include
<sev> <owner-key>...hex encode DH key...</owner-key> <session-info>..hex encode info...</session-info> <policy>...int32 value..</policy> </sev>
if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, libvirt would report an error and refuse to start the guest
3. Libvirt generates the QEMU cli arg to enable SEV using the XML data and starts QEMU, leaving CPUs paused
4. QEMU emits a SEV_MEASURE event containing the measurement blob
Speaking of which, I expect QEMU to have a QMP command to retrieve the measurement, in which case I think libvirt has to provide an API for the user to retrieve the measurement in case libvirtd crashes somewhere between setting up QEMU and waiting for the measurement event from QEMU, or simply because the GO missed the event for some unspecified reason.
Yeah, that's a good point - we also ought to have a pause-reason that reflects that it is paused due to waiting for SEV secrets.
5. Libvirt catches the QEMU event and emits its own VIR_CONNECT_DOMAIN_EVENT_SEV_MEASURE event containing the measurement blob
6. GO does its validation of the measurement
7a If validation failed, then virDomainDestroy() to stop QEMU
7b If validation succeeed
Optionally call
virDomainSetSEVSecret()
Given the fact that we're likely introducing a new <sev> element to the XML config, I'm more inclined to utilizing the existing virSecret interfaces (as was originally suggested) instead of creating a vendor-specific API. You could have an optional secret sub-element within the <sev> element and libvirt would simply check if that secret has a value set, once the GO issues virDomainResume(). Any particular reason for having a specific API for this that I'm missing?
Initially I was intending to suggest extensive use of virSecret, but it turns out that despite being called a "secret", none of the SEV data we are passing around needs protection. Either it is safe to be public, or it is already encrypted. So essentially we just have some data blobs we need to pass into QEMU. I didn't feel we ought to be abusing virSecret as a general purpose mechanism for passing in opaque data blobs which do not need any kind of protection. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Sep 18, 2017 at 11:43:57AM +0200, Erik Skultety wrote:
[...]
> c) what existing communicate interface can be used between libvirt and qemu > to get the measurement ? can we add a new qemu monitor command > 'get_sev_measurement' to get the measurement ? (step 10)
Yes, QMP commands seeem most likely.
> d) how to pass the secret blob from libvirt to qemu ? should we consider > adding a new object (sev-guest-secret) -- libvirt can add the object through > qemu monitor.
Yeah, that looks like a viable option too.
So I could see a flow like the following:
1. mgmt tool calls virConnectGetCapabilities. This returns an XML document that includes the following
<host> ...other bits... <sev> <platform-key>...hex encoded PDH key...</platform-key> </sev> </host>
2. mgmt tool requests to start a guest calling virCreateXML(), passing VIR_DOMAIN_START_PAUSED. The XML would include
<sev> <owner-key>...hex encode DH key...</owner-key> <session-info>..hex encode info...</session-info> <policy>...int32 value..</policy> </sev>
if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, libvirt would report an error and refuse to start the guest For ease of use, I would not add this conditional to libvirt. If <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, I’d just send the "GO" command as it would naturally occur. Unless that would confuse things inside libvirt or QEMU in relation to
On 9/18/17 4:47 AM, Daniel P. Berrange wrote: the measurement and secret… Many of our existing tests focus on other aspects of SEV functionality and so they skip the MEASURE/SECRET phase of launch and just go immediately from LAUNCH_UPDATE_DATA (or VMSA) to LAUNCH_FINISH. I guess the key question will be how will QEMU know when to get the MEASUREMENT and wait for a LAUNCH_SECRET before doing a LAUNCH_FINISH when connected to libvirt. Brijesh, this is your area. It feels to me like QEMU will have to wait to do the LAUNCH_FINISH until it gets the first “go” from libvirt. If that’s right, and assuming the same “go” comes from libvirt with or without VIR_DOMAIN_START_PAUSED, then I’d simply exclude the conditional check. QEMU would get the measurement when it is done sending the data. Though in “real world” uses, I think the conditional is perfectly OK.
3. Libvirt generates the QEMU cli arg to enable SEV using the XML data and starts QEMU, leaving CPUs paused
4. QEMU emits a SEV_MEASURE event containing the measurement blob
Speaking of which, I expect QEMU to have a QMP command to retrieve the measurement, in which case I think libvirt has to provide an API for the user to retrieve the measurement in case libvirtd crashes somewhere between setting up QEMU and waiting for the measurement event from QEMU, or simply because the GO missed the event for some unspecified reason. Yeah, that's a good point - we also ought to have a pause-reason that reflects that it is paused due to waiting for SEV secrets.
5. Libvirt catches the QEMU event and emits its own VIR_CONNECT_DOMAIN_EVENT_SEV_MEASURE event containing the measurement blob
6. GO does its validation of the measurement
7a If validation failed, then virDomainDestroy() to stop QEMU
7b If validation succeeed
Optionally call
virDomainSetSEVSecret() Given the fact that we're likely introducing a new <sev> element to the XML config, I'm more inclined to utilizing the existing virSecret interfaces (as was originally suggested) instead of creating a vendor-specific API. You could have an optional secret sub-element within the <sev> element and libvirt would simply check if that secret has a value set, once the GO issues virDomainResume(). Any particular reason for having a specific API for this that I'm missing? Initially I was intending to suggest extensive use of virSecret, but it turns out that despite being called a "secret", none of the SEV data we are passing around needs protection. Either it is safe to be public, or it is already encrypted. So essentially we just have some data blobs we need to pass into QEMU. I didn't feel we ought to be abusing virSecret as a general purpose mechanism for passing in opaque data blobs which do not need any kind of protection. All of the above looks really good to me. While I agree with Daniel’s analysis of the need for “secret”, I do like using virSecret to convey the notion of secrecy. But it isn’t necessary. The end points are the SEV FW and the guest owner and all secrets they share are already encrypted. Embedding it in the “GO” command feels equally OK to me. Note that sending a secret with a “GO” other than the first one is an error… I don’t think libvirt needs to catch that, though. The SEV FW will.
Regards, Daniel Thanks, Richard
On Mon, Sep 18, 2017 at 07:41:09AM -0500, Richard Relph wrote:
On Mon, Sep 18, 2017 at 11:43:57AM +0200, Erik Skultety wrote:
[...]
> c) what existing communicate interface can be used between libvirt and qemu > to get the measurement ? can we add a new qemu monitor command > 'get_sev_measurement' to get the measurement ? (step 10)
Yes, QMP commands seeem most likely.
> d) how to pass the secret blob from libvirt to qemu ? should we consider > adding a new object (sev-guest-secret) -- libvirt can add the object through > qemu monitor.
Yeah, that looks like a viable option too.
So I could see a flow like the following:
1. mgmt tool calls virConnectGetCapabilities. This returns an XML document that includes the following
<host> ...other bits... <sev> <platform-key>...hex encoded PDH key...</platform-key> </sev> </host>
2. mgmt tool requests to start a guest calling virCreateXML(), passing VIR_DOMAIN_START_PAUSED. The XML would include
<sev> <owner-key>...hex encode DH key...</owner-key> <session-info>..hex encode info...</session-info> <policy>...int32 value..</policy> </sev>
if <sev> is provided and VIR_DOMAIN_START_PAUSED is missing, libvirt would report an error and refuse to start the guest For ease of use, I would not add this conditional to libvirt. If <sev> is
On 9/18/17 4:47 AM, Daniel P. Berrange wrote: provided and VIR_DOMAIN_START_PAUSED is missing, I’d just send the "GO"
I also feel that the presence of the <sev> element might determine the usage of the VIR_DOMAIN_START_PAUSED flag implicitly.
command as it would naturally occur. Unless that would confuse things inside libvirt or QEMU in relation to the measurement and secret… Many of our existing tests focus on other aspects of SEV functionality and so they skip the MEASURE/SECRET phase of launch and just go immediately from LAUNCH_UPDATE_DATA (or VMSA) to LAUNCH_FINISH. I guess the key question will be how will QEMU know when to get the MEASUREMENT and wait for a LAUNCH_SECRET before doing a LAUNCH_FINISH when connected to libvirt. Brijesh, this is your area. It feels to me like QEMU will have to wait to do the LAUNCH_FINISH until it gets the first “go” from libvirt. If that’s right, and assuming the same “go” comes from libvirt with or without VIR_DOMAIN_START_PAUSED, then I’d simply exclude the conditional check. QEMU would get the measurement when it is done sending the data. Though in “real world” uses, I think the conditional is perfectly OK.
3. Libvirt generates the QEMU cli arg to enable SEV using the XML data and starts QEMU, leaving CPUs paused
4. QEMU emits a SEV_MEASURE event containing the measurement blob
Speaking of which, I expect QEMU to have a QMP command to retrieve the measurement, in which case I think libvirt has to provide an API for the user to retrieve the measurement in case libvirtd crashes somewhere between setting up QEMU and waiting for the measurement event from QEMU, or simply because the GO missed the event for some unspecified reason. Yeah, that's a good point - we also ought to have a pause-reason that reflects that it is paused due to waiting for SEV secrets.
5. Libvirt catches the QEMU event and emits its own VIR_CONNECT_DOMAIN_EVENT_SEV_MEASURE event containing the measurement blob
6. GO does its validation of the measurement
7a If validation failed, then virDomainDestroy() to stop QEMU
7b If validation succeeed
Optionally call
virDomainSetSEVSecret() Given the fact that we're likely introducing a new <sev> element to the XML config, I'm more inclined to utilizing the existing virSecret interfaces (as was originally suggested) instead of creating a vendor-specific API. You could have an optional secret sub-element within the <sev> element and libvirt would simply check if that secret has a value set, once the GO issues virDomainResume(). Any particular reason for having a specific API for this that I'm missing? Initially I was intending to suggest extensive use of virSecret, but it turns out that despite being called a "secret", none of the SEV data we are passing around needs protection. Either it is safe to be public, or it is already encrypted. So essentially we just have some data blobs we need to pass into QEMU. I didn't feel we ought to be abusing virSecret as a general purpose mechanism for passing in opaque data blobs which do not need any kind of protection. All of the above looks really good to me. While I agree with Daniel’s analysis of the need for “secret”, I do like using virSecret to convey the notion of secrecy. But it isn’t necessary. The end points are the SEV FW and the guest owner and all secrets they share are already encrypted. Embedding it in the “GO” command feels equally OK to me. Note that sending a secret with a “GO” other than the first one is an error… I don’t think libvirt needs to catch that, though. The SEV FW will.
Setting a secret and resuming a domain are two separate actions, as resuming a domain doesn't take any arbitrary data (ambiguity of the opaque data). This would especially be an issue if we allowed multiple secrets to be passed to the guest. Therefore such a check is transparent to libvirt, as libvirt can't reliably tell how many secrets are there necessary to be passed to the guest (only guest owner knows that) - libvirt would only fail setting the same secret multiple times with QEMU error which in turn would get it from SEV FW). Additionally, virDomainResume() can be called multiple times on the same domain safely, since the domain to be resumed is going to be in the 'running' in all but the first time you call the function. Erik
On Fri, Sep 08, 2017 at 06:57:30AM -0500, Brijesh Singh wrote:
Hi All,
Sorry if below comment doesn't make sense, I might be misunderstanding something basic about SEV. Also sorry about the delay, I've been on vacation.
(sorry for the long message)
CPUs from AMD EPYC family supports Secure Encrypted Virtualization (SEV) feature - the feature allows running encrypted VMs. To enable the feature, I have been submitting patches to Linux kernel [1], Qemu [2] and OVMF [3]. We have been making some good progress in getting patches accepted upstream in Linux and OVMF trees. SEV builds upon SME (Secure Memory Encryption) feature -- SME support just got pulled into 4.14 merge window. The base SEV patches are accepted in OVMF tree -- now we have SEV aware guest BIOS. I am getting ready to take off "RFC" tag from remaining patches to get them reviewed and accepted.
The boot flow for launching an SEV guest is a bit different from a typical guest launch. In order to launch SEV guest from virt-manager or other high-level VM management tools, we need to design and implement new interface between libvirt and qemu, and probably add new APIs in libvirt to be used by VM management tools. I am new to the libvirt and need some expert advice while designing this interface. A pictorial representation for a SEV guest launch flow is available in SEV Spec Appendix A [4].
A typical flow looks like this:
1. Guest owner (GO) asks the cloud provider to launch SEV guest. 2. VM tool asks libvirt to provide its Platform Diffie-Hellman (PDH) key. 3. libvirt opens /dev/sev device to get its PDH and return the blob to the caller. 4. VM tool gives its PDH to GO. 5. GO provides its DH key, session-info and guest policy. 6. VM tool somehow communicates the GO provided information to libvirt. 7. libvirt adds "sev-guest" object in its xml file with all the information obtained from #5
(currently my xml file looks like this)
<qemu:arg value='-object'> <qemu:arg value='sev-guest,id=sev0,policy=<GO_policy>,dh-key-file=<filename>,session-file=<filename>/> <qemu:arg value='-machine'/> <qemu:arg value='memory-encryption=sev0'/>
8. libvirt launches the guest with "-S" 9. While creating the SEV guest qemu does the following i) create encryption context using GO's DH, session-info and guest policy (LAUNCH_START) ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement
This part troubles me. This seems to mean that the guest being launched must know what the measurement of the bios is going to be. This means that the cloud provider can not update the bios without breaking guests. Also, while in practice you typically can run an old bios image on a new qemu instance, this is not really tested so would be very hard to support properly in QEMU. And this looks like a fundamental problem with the hash based measurement that's in hardware. So below I suggest that we layer some software on top to rely on the hash as little as possible.
10. By some interface we must propagate the measurement all the way to GO before libvirt starts the guest. 11. GO verifies the measurement and if measurement matches then it may give a secret blob -- which must be injected into the guest before libvirt starts the VM. If verification failed, GO will request cloud provider to destroy the VM. 12. After secret blob is injected into guest, we call LAUNCH_FINISH to destory the encryption context. 13. libvirt issues "continue" command to resume the guest boot.
Please note that the measurement value is protected with transport encryption key (TIK) and it changes on each run. Similarly the secret blob provided by GO does not need to be protected using libvirt/qemu APIs. The secret is protected by TIK. From qemu and libvirt point of view these are blobs and must be passed as-is to the SEV FW.
So here's an alternative idea for starting guests: How about building a minimal shim firmware that runs on a single CPU and uses no hardware at all, it just contains the secret blob. That firmware just immediately stops and signals hypervisor that it is ready to be run in the cloud. Have user generate and start this shim firmware as a guest in a private setup, then export it out using SEND_* commands. Then instead of asking to launch guest, you ask provider to load it with RECEIVE_* commands. Unlike bios the shim firmware can hopefully be static so supporting it across qemu versions should be easy. The shim firmware then loads bios from qemu, verifies it in any way it sees fit (e.g. it could check a signature, version, etc: it is not limited to a hardware hash anymore). It then jumps to the bios. While not exactly the same, there is some similarity here with how people solved the issues around secureboot - by using a shim. Thanks!
Questions: a) Do we need to add a new set of APIs in libvirt to return the PDH from libvirt and VM tool ? Or can we use some pre-existing APIs to pass the opaque blobs ? (this is mainly for step 3 and 6) b) do we need to define a new xml tag to for memory-encryption ? or just use the qemu:args tag ? (step 6) c) what existing communicate interface can be used between libvirt and qemu to get the measurement ? can we add a new qemu monitor command 'get_sev_measurement' to get the measurement ? (step 10) d) how to pass the secret blob from libvirt to qemu ? should we consider adding a new object (sev-guest-secret) -- libvirt can add the object through qemu monitor.
[1] https://marc.info/?l=kvm&m=150092661105069&w=2 [2] https://marc.info/?l=qemu-devel&m=148901186615642&w=2 [3] https://lists.01.org/pipermail/edk2-devel/2017-July/012220.html [4] http://support.amd.com/TechDocs/55766_SEV-KM%20API_Specification.pdf
Thanks
Brijesh
* Michael S. Tsirkin (mst@redhat.com) wrote:
On Fri, Sep 08, 2017 at 06:57:30AM -0500, Brijesh Singh wrote:
Hi All,
Sorry if below comment doesn't make sense, I might be misunderstanding something basic about SEV. Also sorry about the delay, I've been on vacation.
(sorry for the long message)
CPUs from AMD EPYC family supports Secure Encrypted Virtualization (SEV) feature - the feature allows running encrypted VMs. To enable the feature, I have been submitting patches to Linux kernel [1], Qemu [2] and OVMF [3]. We have been making some good progress in getting patches accepted upstream in Linux and OVMF trees. SEV builds upon SME (Secure Memory Encryption) feature -- SME support just got pulled into 4.14 merge window. The base SEV patches are accepted in OVMF tree -- now we have SEV aware guest BIOS. I am getting ready to take off "RFC" tag from remaining patches to get them reviewed and accepted.
The boot flow for launching an SEV guest is a bit different from a typical guest launch. In order to launch SEV guest from virt-manager or other high-level VM management tools, we need to design and implement new interface between libvirt and qemu, and probably add new APIs in libvirt to be used by VM management tools. I am new to the libvirt and need some expert advice while designing this interface. A pictorial representation for a SEV guest launch flow is available in SEV Spec Appendix A [4].
A typical flow looks like this:
1. Guest owner (GO) asks the cloud provider to launch SEV guest. 2. VM tool asks libvirt to provide its Platform Diffie-Hellman (PDH) key. 3. libvirt opens /dev/sev device to get its PDH and return the blob to the caller. 4. VM tool gives its PDH to GO. 5. GO provides its DH key, session-info and guest policy. 6. VM tool somehow communicates the GO provided information to libvirt. 7. libvirt adds "sev-guest" object in its xml file with all the information obtained from #5
(currently my xml file looks like this)
<qemu:arg value='-object'> <qemu:arg value='sev-guest,id=sev0,policy=<GO_policy>,dh-key-file=<filename>,session-file=<filename>/> <qemu:arg value='-machine'/> <qemu:arg value='memory-encryption=sev0'/>
8. libvirt launches the guest with "-S" 9. While creating the SEV guest qemu does the following i) create encryption context using GO's DH, session-info and guest policy (LAUNCH_START) ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement
This part troubles me. This seems to mean that the guest being launched must know what the measurement of the bios is going to be. This means that the cloud provider can not update the bios without breaking guests. Also, while in practice you typically can run an old bios image on a new qemu instance, this is not really tested so would be very hard to support properly in QEMU.
And this looks like a fundamental problem with the hash based measurement that's in hardware. So below I suggest that we layer some software on top to rely on the hash as little as possible.
I think the normal way to solve this is that the Distro would provide a list of the signatures, and the GO would check the bios measurement against that list. The GO needs to keep that list up to date; either directly downloading it from the Distro or a copy signed by the Distro. Dave
10. By some interface we must propagate the measurement all the way to GO before libvirt starts the guest. 11. GO verifies the measurement and if measurement matches then it may give a secret blob -- which must be injected into the guest before libvirt starts the VM. If verification failed, GO will request cloud provider to destroy the VM. 12. After secret blob is injected into guest, we call LAUNCH_FINISH to destory the encryption context. 13. libvirt issues "continue" command to resume the guest boot.
Please note that the measurement value is protected with transport encryption key (TIK) and it changes on each run. Similarly the secret blob provided by GO does not need to be protected using libvirt/qemu APIs. The secret is protected by TIK. From qemu and libvirt point of view these are blobs and must be passed as-is to the SEV FW.
So here's an alternative idea for starting guests:
How about building a minimal shim firmware that runs on a single CPU and uses no hardware at all, it just contains the secret blob.
That firmware just immediately stops and signals hypervisor that it is ready to be run in the cloud.
Have user generate and start this shim firmware as a guest in a private setup, then export it out using SEND_* commands.
Then instead of asking to launch guest, you ask provider to load it with RECEIVE_* commands.
Unlike bios the shim firmware can hopefully be static so supporting it across qemu versions should be easy.
The shim firmware then loads bios from qemu, verifies it in any way it sees fit (e.g. it could check a signature, version, etc: it is not limited to a hardware hash anymore). It then jumps to the bios.
While not exactly the same, there is some similarity here with how people solved the issues around secureboot - by using a shim.
Thanks!
Questions: a) Do we need to add a new set of APIs in libvirt to return the PDH from libvirt and VM tool ? Or can we use some pre-existing APIs to pass the opaque blobs ? (this is mainly for step 3 and 6) b) do we need to define a new xml tag to for memory-encryption ? or just use the qemu:args tag ? (step 6) c) what existing communicate interface can be used between libvirt and qemu to get the measurement ? can we add a new qemu monitor command 'get_sev_measurement' to get the measurement ? (step 10) d) how to pass the secret blob from libvirt to qemu ? should we consider adding a new object (sev-guest-secret) -- libvirt can add the object through qemu monitor.
[1] https://marc.info/?l=kvm&m=150092661105069&w=2 [2] https://marc.info/?l=qemu-devel&m=148901186615642&w=2 [3] https://lists.01.org/pipermail/edk2-devel/2017-July/012220.html [4] http://support.amd.com/TechDocs/55766_SEV-KM%20API_Specification.pdf
Thanks
Brijesh
-- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
Hi Michael, On 09/26/2017 09:36 AM, Michael S. Tsirkin wrote: ...
8. libvirt launches the guest with "-S" 9. While creating the SEV guest qemu does the following i) create encryption context using GO's DH, session-info and guest policy (LAUNCH_START) ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement
This part troubles me. This seems to mean that the guest being launched must know what the measurement of the bios is going to be. This means that the cloud provider can not update the bios without breaking guests. Also, while in practice you typically can run an old bios image on a new qemu instance, this is not really tested so would be very hard to support properly in QEMU.
The guest itself does not need to know the measurement of the bios -- the SEV launch flow empowers the GO to validate the bootstrap code (bios) before GO can provide a confidential information to the guest. Please note that the validating of the measurement flow is optional. GO can ask cloud provider to ignore the measurement all together and boot the SEV guest. The measurement flow can be useful when GO decides to provide a custom bios and want to know that his bios is used for booting the guest. In this case, since the guest owner knows the initial contents of the guest at boot, he can request the measurement from the cloud provider and compare it with what the guest owner expects.
And this looks like a fundamental problem with the hash based measurement that's in hardware. So below I suggest that we layer some software on top to rely on the hash as little as possible.
10. By some interface we must propagate the measurement all the way to GO before libvirt starts the guest. 11. GO verifies the measurement and if measurement matches then it may give a secret blob -- which must be injected into the guest before libvirt starts the VM. If verification failed, GO will request cloud provider to destroy the VM. 12. After secret blob is injected into guest, we call LAUNCH_FINISH to destory the encryption context. 13. libvirt issues "continue" command to resume the guest boot.
Please note that the measurement value is protected with transport encryption key (TIK) and it changes on each run. Similarly the secret blob provided by GO does not need to be protected using libvirt/qemu APIs. The secret is protected by TIK. From qemu and libvirt point of view these are blobs and must be passed as-is to the SEV FW.
So here's an alternative idea for starting guests:
How about building a minimal shim firmware that runs on a single CPU and uses no hardware at all, it just contains the secret blob.
That firmware just immediately stops and signals hypervisor that it is ready to be run in the cloud.
Have user generate and start this shim firmware as a guest in a private setup, then export it out using SEND_* commands.
Then instead of asking to launch guest, you ask provider to load it with RECEIVE_* commands.
Unlike bios the shim firmware can hopefully be static so supporting it across qemu versions should be easy.
The shim firmware then loads bios from qemu, verifies it in any way it sees fit (e.g. it could check a signature, version, etc: it is not limited to a hardware hash anymore). It then jumps to the bios.
While not exactly the same, there is some similarity here with how people solved the issues around secureboot - by using a shim.
On Wed, Sep 27, 2017 at 08:39:24AM -0500, Brijesh Singh wrote:
Hi Michael,
On 09/26/2017 09:36 AM, Michael S. Tsirkin wrote:
...
8. libvirt launches the guest with "-S" 9. While creating the SEV guest qemu does the following i) create encryption context using GO's DH, session-info and guest policy (LAUNCH_START) ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement
This part troubles me. This seems to mean that the guest being launched must know what the measurement of the bios is going to be. This means that the cloud provider can not update the bios without breaking guests. Also, while in practice you typically can run an old bios image on a new qemu instance, this is not really tested so would be very hard to support properly in QEMU.
The guest itself does not need to know the measurement of the bios -- the SEV launch flow empowers the GO to validate the bootstrap code (bios) before GO can provide a confidential information to the guest. Please note that the validating of the measurement flow is optional. GO can ask cloud provider to ignore the measurement all together and boot the SEV guest.
As the OS runs on top of the bios, it does not seem prudent to boot the SEV guest without a way to verify that the bios is safe to use. Linux generally trusts the firmware it runs on. Are you looking for examples of how a malicious firmware can leak info out of the guest?
The measurement flow can be useful when GO decides to provide a custom bios and want to know that his bios is used for booting the guest. In this case, since the guest owner knows the initial contents of the guest at boot, he can request the measurement from the cloud provider and compare it with what the guest owner expects.
I agree the measurement works for this, but If that's the only case, I'd say it's not all that interesting since most people will use a standard bios. I do think something needs to vaidate the bios though, and I do not think naively measuring the hash of the full bios is a way to do this that we can support well long term.
And this looks like a fundamental problem with the hash based measurement that's in hardware. So below I suggest that we layer some software on top to rely on the hash as little as possible.
10. By some interface we must propagate the measurement all the way to GO before libvirt starts the guest. 11. GO verifies the measurement and if measurement matches then it may give a secret blob -- which must be injected into the guest before libvirt starts the VM. If verification failed, GO will request cloud provider to destroy the VM. 12. After secret blob is injected into guest, we call LAUNCH_FINISH to destory the encryption context. 13. libvirt issues "continue" command to resume the guest boot.
Please note that the measurement value is protected with transport encryption key (TIK) and it changes on each run. Similarly the secret blob provided by GO does not need to be protected using libvirt/qemu APIs. The secret is protected by TIK. From qemu and libvirt point of view these are blobs and must be passed as-is to the SEV FW.
So here's an alternative idea for starting guests:
How about building a minimal shim firmware that runs on a single CPU and uses no hardware at all, it just contains the secret blob.
That firmware just immediately stops and signals hypervisor that it is ready to be run in the cloud.
Have user generate and start this shim firmware as a guest in a private setup, then export it out using SEND_* commands.
Then instead of asking to launch guest, you ask provider to load it with RECEIVE_* commands.
Unlike bios the shim firmware can hopefully be static so supporting it across qemu versions should be easy.
The shim firmware then loads bios from qemu, verifies it in any way it sees fit (e.g. it could check a signature, version, etc: it is not limited to a hardware hash anymore). It then jumps to the bios.
While not exactly the same, there is some similarity here with how people solved the issues around secureboot - by using a shim.
Forgive the top post... some of the conversation has been trimmed, but I need to go back to first principles of SEV in order to make sure we all have a clear understanding of what the goal is. The goal - for BOTH guest owner and cloud provider - is to get to a VM where ONLY the guest owner (GO) has access to the GO "secrets". Legitimate cloud providers (ie, those that wish to not retain a back door in to their customer's VMs) want this every bit as much as GOs want it. It is this privacy concern that some believe holds back broader adoption of public cloud for sensitive applications. To provide this additional privacy will require some changes to the "untrustable" model that seems to be in place now. There is value for everyone in creating a "trustable" model. Given that, the root of the problem for the GO is trust. How can the GO know that every instruction in their VM is "theirs"? AMD Epyc CPUs decrypt every instruction fetch (and guest page table walk) in an SEV guest VM with that VM's random memory encryption key. (Data can either be encrypted or not, at the guest's choosing.) Only the SEV FW and the guest itself can encrypt memory with that key. The SEV FW measures every byte it encrypts for the guest and provides that measurement to the GO. The GO is free to ignore the value and run the guest "as-is". But recommended practice will be to inspect the measurement, verify it, and only then provide the guest VM with "secrets" necessary to decrypt disks, connect to privileged network resources, etc. As has been observed, the BIOS has a great deal of power. It is impossible to maintain the GO's privacy in a VM where the BIOS (or any other code, for that matter) is "unknown". It simply is a violation of the trust model both the GO and the CP want to have to allow unknown code from the CP to enter the GO's VM. (Yes, there are LOTS of other ways for untrusted code to get in and secrets to get out... we're only trying to close this door between the CP and the guest VM at this time. ;-) I anticipate that legitimate cloud providers will be happy (or at least willing) to share with customers the source for their BIOS. The GO can inspect the source, build the binary from that source, and generate the required hash. Or they may just trust that someone else has done that work and accept the hash the CP posts on their BIOS image. (Note that when the hash is returned by the SEV FW, it is in HMAC form, with a nonce that the GO can compute, and a key the GO provided at launch time.) Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, or even a BIOS+kernel+initrd is really not too significant. What is significant is that the GO has a basis for trusting all code that is imported in to their VM by the CP. And that NONE of the code provided by the CP is "unknown" and unauditable by the GO. If the CP has a way to inject code unknown to the GO in to the guest VM, the trust model is broken and both GO and CP suffer the consequences. When the CP needs to update the BIOS image, they will have to inform the GO and allow the GO to establish trust in the CP's new BIOS image somehow. I hope that helps outline what we're doing, and why. Richard On 9/27/17 11:12 AM, Michael S. Tsirkin wrote:
On Wed, Sep 27, 2017 at 08:39:24AM -0500, Brijesh Singh wrote:
Hi Michael,
On 09/26/2017 09:36 AM, Michael S. Tsirkin wrote:
...
8. libvirt launches the guest with "-S" 9. While creating the SEV guest qemu does the following i) create encryption context using GO's DH, session-info and guest policy (LAUNCH_START) ii) encrypts the guest bios (LAUNCH_UPDATE_DATA) iii) calls LAUNCH_MEASUREMENT to get the encrypted bios measurement
This part troubles me. This seems to mean that the guest being launched must know what the measurement of the bios is going to be. This means that the cloud provider can not update the bios without breaking guests. Also, while in practice you typically can run an old bios image on a new qemu instance, this is not really tested so would be very hard to support properly in QEMU.
The guest itself does not need to know the measurement of the bios -- the SEV launch flow empowers the GO to validate the bootstrap code (bios) before GO can provide a confidential information to the guest. Please note that the validating of the measurement flow is optional. GO can ask cloud provider to ignore the measurement all together and boot the SEV guest.
As the OS runs on top of the bios, it does not seem prudent to boot the SEV guest without a way to verify that the bios is safe to use. Linux generally trusts the firmware it runs on. Are you looking for examples of how a malicious firmware can leak info out of the guest?
The measurement flow can be useful when GO decides to provide a custom bios and want to know that his bios is used for booting the guest. In this case, since the guest owner knows the initial contents of the guest at boot, he can request the measurement from the cloud provider and compare it with what the guest owner expects.
I agree the measurement works for this, but If that's the only case, I'd say it's not all that interesting since most people will use a standard bios.
I do think something needs to vaidate the bios though, and I do not think naively measuring the hash of the full bios is a way to do this that we can support well long term.
And this looks like a fundamental problem with the hash based measurement that's in hardware. So below I suggest that we layer some software on top to rely on the hash as little as possible.
10. By some interface we must propagate the measurement all the way to GO before libvirt starts the guest. 11. GO verifies the measurement and if measurement matches then it may give a secret blob -- which must be injected into the guest before libvirt starts the VM. If verification failed, GO will request cloud provider to destroy the VM. 12. After secret blob is injected into guest, we call LAUNCH_FINISH to destory the encryption context. 13. libvirt issues "continue" command to resume the guest boot.
Please note that the measurement value is protected with transport encryption key (TIK) and it changes on each run. Similarly the secret blob provided by GO does not need to be protected using libvirt/qemu APIs. The secret is protected by TIK. From qemu and libvirt point of view these are blobs and must be passed as-is to the SEV FW.
So here's an alternative idea for starting guests:
How about building a minimal shim firmware that runs on a single CPU and uses no hardware at all, it just contains the secret blob.
That firmware just immediately stops and signals hypervisor that it is ready to be run in the cloud.
Have user generate and start this shim firmware as a guest in a private setup, then export it out using SEND_* commands.
Then instead of asking to launch guest, you ask provider to load it with RECEIVE_* commands.
Unlike bios the shim firmware can hopefully be static so supporting it across qemu versions should be easy.
The shim firmware then loads bios from qemu, verifies it in any way it sees fit (e.g. it could check a signature, version, etc: it is not limited to a hardware hash anymore). It then jumps to the bios.
While not exactly the same, there is some similarity here with how people solved the issues around secureboot - by using a shim.
On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote:
Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, or even a BIOS+kernel+initrd is really not too significant. What is significant is that the GO has a basis for trusting all code that is imported in to their VM by the CP. And that NONE of the code provided by the CP is "unknown" and unauditable by the GO. If the CP has a way to inject code unknown to the GO in to the guest VM, the trust model is broken and both GO and CP suffer the consequences.
Absolutely.
When the CP needs to update the BIOS image, they will have to inform the GO and allow the GO to establish trust in the CP's new BIOS image somehow.
This GO update on every BIOS change is imho is not a workable model. You want something like checking the BIOS signature instead. And since hardware is all hash based, you need the shim to do it in software. -- MST
On 9/29/17 2:34 PM, Michael S. Tsirkin wrote:
On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote:
Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, or even a BIOS+kernel+initrd is really not too significant. What is significant is that the GO has a basis for trusting all code that is imported in to their VM by the CP. And that NONE of the code provided by the CP is "unknown" and unauditable by the GO. If the CP has a way to inject code unknown to the GO in to the guest VM, the trust model is broken and both GO and CP suffer the consequences.
Absolutely.
When the CP needs to update the BIOS image, they will have to inform the GO and allow the GO to establish trust in the CP's new BIOS image somehow.
This GO update on every BIOS change is imho is not a workable model. You want something like checking the BIOS signature instead. And since hardware is all hash based, you need the shim to do it in software.
A BIOS "signed" by the CP doesn't meet the security requirement. It is code that is "unknown" to the GO. The (legitimate) CP does NOT want to be in that position of trust. If they are, then some government somewhere is going to insist that they sign a BIOS that allows the government to spy on the GO's VMs, and steal secrets from it. Or some hacker admin will do it "for fun". How often do large public CPs really change their BIOSes? My sense is that large public CPs prefer stability over "latest and greatest". And, perhaps more importantly, if a CP are able to sell a "more secure" VM, one that justifies a higher price per vCPU hour, wouldn't that warrant some changes in the "insecure" model being used today? Richard
On 9/29/17 2:48 PM, Richard Relph wrote:
On 9/29/17 2:34 PM, Michael S. Tsirkin wrote:
On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote:
Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, or even a BIOS+kernel+initrd is really not too significant. What is significant is that the GO has a basis for trusting all code that is imported in to their VM by the CP. And that NONE of the code provided by the CP is "unknown" and unauditable by the GO. If the CP has a way to inject code unknown to the GO in to the guest VM, the trust model is broken and both GO and CP suffer the consequences.
Absolutely.
When the CP needs to update the BIOS image, they will have to inform the GO and allow the GO to establish trust in the CP's new BIOS image somehow.
This GO update on every BIOS change is imho is not a workable model. You want something like checking the BIOS signature instead. And since hardware is all hash based, you need the shim to do it in software.
A BIOS "signed" by the CP doesn't meet the security requirement. It is code that is "unknown" to the GO.
The (legitimate) CP does NOT want to be in that position of trust. If they are, then some government somewhere is going to insist that they sign a BIOS that allows the government to spy on the GO's VMs, and steal secrets from it. Or some hacker admin will do it "for fun".
How often do large public CPs really change their BIOSes? My sense is that large public CPs prefer stability over "latest and greatest".
And, perhaps more importantly, if a CP are able to sell a "more secure" VM, one that justifies a higher price per vCPU hour, wouldn't that warrant some changes in the "insecure" model being used today?
Ultimately, I think both approaches are "doable". It will be a CP and GO decision. If the GO trusts the CP, the shim+signed BIOS will work fine. If GO requires a more secure VM and the CP wants to offer it, the CP will figure out a way to satisfy the GO's "trust issue" that the BIOS can't be used to circumvent SEV's protections. Depending on your level of paranoia, that may require advance notice of BIOS changes, or even allowing the GO to provide the BIOS themselves, written to a spec supported by the CP's HV, and/or based on BIOS code provided by the CP. It's a business decision and I think SEV can support both. That said, AMD currently has no plans to write a shim that can verify the signature on a CP-provided BIOS image. Richard
Richard
On Fri, Sep 29, 2017 at 03:07:40PM -0500, Richard Relph wrote:
On 9/29/17 2:48 PM, Richard Relph wrote:
On 9/29/17 2:34 PM, Michael S. Tsirkin wrote:
On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote:
Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, or even a BIOS+kernel+initrd is really not too significant. What is significant is that the GO has a basis for trusting all code that is imported in to their VM by the CP. And that NONE of the code provided by the CP is "unknown" and unauditable by the GO. If the CP has a way to inject code unknown to the GO in to the guest VM, the trust model is broken and both GO and CP suffer the consequences.
Absolutely.
When the CP needs to update the BIOS image, they will have to inform the GO and allow the GO to establish trust in the CP's new BIOS image somehow.
This GO update on every BIOS change is imho is not a workable model. You want something like checking the BIOS signature instead. And since hardware is all hash based, you need the shim to do it in software.
A BIOS "signed" by the CP doesn't meet the security requirement. It is code that is "unknown" to the GO.
The (legitimate) CP does NOT want to be in that position of trust. If they are, then some government somewhere is going to insist that they sign a BIOS that allows the government to spy on the GO's VMs, and steal secrets from it. Or some hacker admin will do it "for fun".
How often do large public CPs really change their BIOSes? My sense is that large public CPs prefer stability over "latest and greatest".
And, perhaps more importantly, if a CP are able to sell a "more secure" VM, one that justifies a higher price per vCPU hour, wouldn't that warrant some changes in the "insecure" model being used today?
Ultimately, I think both approaches are "doable". It will be a CP and GO decision. If the GO trusts the CP, the shim+signed BIOS will work fine.
I think there's a misunderstanding. A trusted software vendor would sign the BIOS. GO would verify it. Trusting the CP is not required.
If GO requires a more secure VM and the CP wants to offer it, the CP will figure out a way to satisfy the GO's "trust issue" that the BIOS can't be used to circumvent SEV's protections. Depending on your level of paranoia, that may require advance notice of BIOS changes, or even allowing the GO to provide the BIOS themselves, written to a spec supported by the CP's HV, and/or based on BIOS code provided by the CP.
We are discussing this on a qemu mailing list, aren't we? And from QEMU point of view, I think it won't be able to support a requirement to boot ancient bios versions on new machine types indefinitely with good performance and also fix security issues in them in a timely manner somehow.
It's a business decision and I think SEV can support both. That said, AMD currently has no plans to write a shim that can verify the signature on a CP-provided BIOS image.
Richard
Someone else will have to work on a supportable solution for QEMU then.
Richard
On Fri, Sep 29, 2017 at 03:07:40PM -0500, Richard Relph wrote:
Depending on your level of paranoia, that may require advance notice of BIOS changes, or even allowing the GO to provide the BIOS themselves, written to a spec supported by the CP's HV, and/or based on BIOS code provided by the CP.
BTW this last most secure option is easy to implement with the shim because the shim is using very little in terms of the HV interface. User can then easily build it from source. -- MST
On Fri, Sep 29, 2017 at 03:07:40PM -0500, Richard Relph wrote:
It's a business decision and I think SEV can support both.
I think what has been missed in the noise is the fact that with VM launch, key distribution is a huge problem. With the shim the key distribution problem can go completely away, as you just start it in the private cloud and include the key with the shim, then use the send/recv machinery to migrate to the public one. -- MST
On Fri, Sep 29, 2017 at 02:48:45PM -0500, Richard Relph wrote:
On 9/29/17 2:34 PM, Michael S. Tsirkin wrote:
On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote:
Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, or even a BIOS+kernel+initrd is really not too significant. What is significant is that the GO has a basis for trusting all code that is imported in to their VM by the CP. And that NONE of the code provided by the CP is "unknown" and unauditable by the GO. If the CP has a way to inject code unknown to the GO in to the guest VM, the trust model is broken and both GO and CP suffer the consequences.
Absolutely.
When the CP needs to update the BIOS image, they will have to inform the GO and allow the GO to establish trust in the CP's new BIOS image somehow.
This GO update on every BIOS change is imho is not a workable model. You want something like checking the BIOS signature instead. And since hardware is all hash based, you need the shim to do it in software.
A BIOS "signed" by the CP doesn't meet the security requirement. It is code that is "unknown" to the GO.
There is a misunderstanding here. BIOS would not be signed by a CP. It would be signed by a trusted software vendor e.g. by Red Hat.
The (legitimate) CP does NOT want to be in that position of trust. If they are, then some government somewhere is going to insist that they sign a BIOS that allows the government to spy on the GO's VMs, and steal secrets from it. Or some hacker admin will do it "for fun".
How often do large public CPs really change their BIOSes? My sense is that large public CPs prefer stability over "latest and greatest".
CPs just do dnf update. Software vendors change BIOSes. And we do change them. Look at number of revisions for seabios in e.g. Fedora. More importantly we might need to change them quickly e.g. because of a security issue. Adding the need to coordinate with all GOs is not going to work. Neither can QEMU support booting old BIOS versions on new machine types indefinitely.
And, perhaps more importantly, if a CP are able to sell a "more secure" VM, one that justifies a higher price per vCPU hour, wouldn't that warrant some changes in the "insecure" model being used today?
Richard
Absolutely. CPs have no business signing images. But it is just not really feasible for software vendors to distribute hashes. -- MST
On 09/29/17 23:16, Michael S. Tsirkin wrote:
On Fri, Sep 29, 2017 at 02:48:45PM -0500, Richard Relph wrote:
On 9/29/17 2:34 PM, Michael S. Tsirkin wrote:
On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote:
Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, or even a BIOS+kernel+initrd is really not too significant. What is significant is that the GO has a basis for trusting all code that is imported in to their VM by the CP. And that NONE of the code provided by the CP is "unknown" and unauditable by the GO. If the CP has a way to inject code unknown to the GO in to the guest VM, the trust model is broken and both GO and CP suffer the consequences.
Absolutely.
When the CP needs to update the BIOS image, they will have to inform the GO and allow the GO to establish trust in the CP's new BIOS image somehow.
This GO update on every BIOS change is imho is not a workable model. You want something like checking the BIOS signature instead. And since hardware is all hash based, you need the shim to do it in software.
A BIOS "signed" by the CP doesn't meet the security requirement. It is code that is "unknown" to the GO.
There is a misunderstanding here.
BIOS would not be signed by a CP. It would be signed by a trusted software vendor e.g. by Red Hat.
The (legitimate) CP does NOT want to be in that position of trust. If they are, then some government somewhere is going to insist that they sign a BIOS that allows the government to spy on the GO's VMs, and steal secrets from it. Or some hacker admin will do it "for fun".
How often do large public CPs really change their BIOSes? My sense is that large public CPs prefer stability over "latest and greatest".
CPs just do dnf update. Software vendors change BIOSes.
And we do change them. Look at number of revisions for seabios in e.g. Fedora. More importantly we might need to change them quickly e.g. because of a security issue. Adding the need to coordinate with all GOs is not going to work. Neither can QEMU support booting old BIOS versions on new machine types indefinitely.
And, perhaps more importantly, if a CP are able to sell a "more secure" VM, one that justifies a higher price per vCPU hour, wouldn't that warrant some changes in the "insecure" model being used today?
Richard
Absolutely. CPs have no business signing images. But it is just not really feasible for software vendors to distribute hashes.
Can this be helped by "reproducible builds"? Like, - guest firmware vendor publishes the source package, - GO asynchronously audits the source package (most likely incrementally, reviewing the new, broken-out patches in the source package), - GO builds the binary package, - GO verifies that it bit-wise matches the guest fw vendor's binary package, - GO adds the fw binary's hash to their own whitelist. By virtue of releasing the source package of the guest firmware (and by announcing it via another erratum), the guest fw vendor implicitly notifies all GOs. It is then up to the individual GOs to evaluate the changes on their own time, and to switch to the new fw binary. It would even be OK if the GO built their own fw binary from the source package, and submitted that to the CP, as part of the guest payload. The guest firmware vendor would still be able to support this build, since it would match the vendor's own binary 100%. Thanks Laszlo
On Sat, Sep 30, 2017 at 12:16:55AM +0300, Michael S. Tsirkin wrote:
On Fri, Sep 29, 2017 at 02:48:45PM -0500, Richard Relph wrote:
On 9/29/17 2:34 PM, Michael S. Tsirkin wrote:
On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote:
Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, or even a BIOS+kernel+initrd is really not too significant. What is significant is that the GO has a basis for trusting all code that is imported in to their VM by the CP. And that NONE of the code provided by the CP is "unknown" and unauditable by the GO. If the CP has a way to inject code unknown to the GO in to the guest VM, the trust model is broken and both GO and CP suffer the consequences.
Absolutely.
When the CP needs to update the BIOS image, they will have to inform the GO and allow the GO to establish trust in the CP's new BIOS image somehow.
This GO update on every BIOS change is imho is not a workable model. You want something like checking the BIOS signature instead. And since hardware is all hash based, you need the shim to do it in software.
A BIOS "signed" by the CP doesn't meet the security requirement. It is code that is "unknown" to the GO.
There is a misunderstanding here.
BIOS would not be signed by a CP. It would be signed by a trusted software vendor e.g. by Red Hat.
The (legitimate) CP does NOT want to be in that position of trust. If they are, then some government somewhere is going to insist that they sign a BIOS that allows the government to spy on the GO's VMs, and steal secrets from it. Or some hacker admin will do it "for fun".
How often do large public CPs really change their BIOSes? My sense is that large public CPs prefer stability over "latest and greatest".
CPs just do dnf update. Software vendors change BIOSes.
No they really don't do that. Cloud providers will stick on a golden image of RHEL. If and when they need to do an upgrade, they won't usually do a 'dnf update' because that imposes risk of breaking VMs on that node with no fallback path. Instead they would usually deploy brand new nodes with the new software version, and then live migrate VMs off the old hosts.
And we do change them. Look at number of revisions for seabios in e.g. Fedora. More importantly we might need to change them quickly e.g. because of a security issue. Adding the need to coordinate with all GOs is not going to work. Neither can QEMU support booting old BIOS versions on new machine types indefinitely.
I don't think Fedora is a good fit for SEV - a long term supported distro like RHEL is the more apt target, and there you are looking at 6 months lifetime, except in rare cases of having a BIOS security flaw to pyush out.
And, perhaps more importantly, if a CP are able to sell a "more secure" VM, one that justifies a higher price per vCPU hour, wouldn't that warrant some changes in the "insecure" model being used today?
Richard
Absolutely. CPs have no business signing images. But it is just not really feasible for software vendors to distribute hashes.
Why can't software vendors distribute hashes - they are best placed to, particularly when a single vendor is supplying the whole stack of base OS, virt and OpenStack as a single coordinated product. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Sep 29, 2017 at 02:48:45PM -0500, Richard Relph wrote:
On 9/29/17 2:34 PM, Michael S. Tsirkin wrote:
On Wed, Sep 27, 2017 at 02:06:10PM -0500, Richard Relph wrote:
Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, or even a BIOS+kernel+initrd is really not too significant. What is significant is that the GO has a basis for trusting all code that is imported in to their VM by the CP. And that NONE of the code provided by the CP is "unknown" and unauditable by the GO. If the CP has a way to inject code unknown to the GO in to the guest VM, the trust model is broken and both GO and CP suffer the consequences.
Absolutely.
When the CP needs to update the BIOS image, they will have to inform the GO and allow the GO to establish trust in the CP's new BIOS image somehow.
This GO update on every BIOS change is imho is not a workable model. You want something like checking the BIOS signature instead. And since hardware is all hash based, you need the shim to do it in software.
A BIOS "signed" by the CP doesn't meet the security requirement. It is code that is "unknown" to the GO.
The (legitimate) CP does NOT want to be in that position of trust. If they are, then some government somewhere is going to insist that they sign a BIOS that allows the government to spy on the GO's VMs, and steal secrets from it. Or some hacker admin will do it "for fun".
How often do large public CPs really change their BIOSes? My sense is that large public CPs prefer stability over "latest and greatest".
It is hard to generalize, but from a RHEL POV, we typically do major updates of the virt stack every ~6 months, and these will include BIOS updates. So if a cloud vendor is following the RHEL update stream actively that's the kind of cadence you'd expect. The gotcha would come if there were out-of-band security updates for BIOS which caused it to be updated before the 6 month window. Fortunately I've not see these happen often, so I don't think its a fatal problem. IOW, I tend to agree with you that this is not really a blocking problem to the use of SEV in cloud.
And, perhaps more importantly, if a CP are able to sell a "more secure" VM, one that justifies a higher price per vCPU hour, wouldn't that warrant some changes in the "insecure" model being used today?
Yes. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Side topic; sorry if it has been mentioned elsewhere: On 09/27/17 21:06, Richard Relph wrote:
Whether the "BIOS" is a "static shim" as Michael suggests, or a full BIOS, or even a BIOS+kernel+initrd is really not too significant. What is significant is that the GO has a basis for trusting all code that is imported in to their VM by the CP. And that NONE of the code provided by the CP is "unknown" and unauditable by the GO. If the CP has a way to inject code unknown to the GO in to the guest VM, the trust model is broken and both GO and CP suffer the consequences.
The expansion ROMs (containing UEFI drivers) of emulated PCI devices, and the same of assigned physical PCI devices, constitute another channel through which code enters the guest from the outside (i.e., from the Cloud Provider). The ROM BARs from which the guest firmware reads the UEFI binaries are not guest RAM, they are MMIO. (For execution, the drivers are copied into encrypted guest RAM.) If the guest has Secure Boot enabled, then the oproms are verified[*] (and not launched if verification fails), but this is slightly different from what I understand under audit-by-GO. It means the GO wouldn't get a measurement of the oproms for one-by-one clearing, when about to green-light a guest startup. Instead the GO would ensure that Secure Boot be enabled with the right certificates (and/or executable hashes) enrolled off the bat, and then implicitly trust all oprom drivers accepted by those certs / hashes. It's another layer of indirection. This is likely nothing new qualitatively, but "the devil is in the details", so I thought it was worth raising. [*] For edk2 / OvmfPkg specifics, I'll mention gEfiSecurityPkgTokenSpaceGuid.PcdOptionRomImageVerificationPolicy The SecurityPkg default is 0x04 ("Deny execution when there is security violation"). However, OVMF sets it to 0x00 ("Always trust the image"). Please see the following commit for the reasons: https://github.com/tianocore/edk2/commit/1fea9ddb4e3fd Brijesh, for SEV guests, we likely want to flip this PCD to 0x04, in the AmdSevInitialize() function, in "OvmfPkg/PlatformPei/AmdSev.c". For that we'll also have to change the PCD from fixed-at-build to dynamic, but that in turn will require a change to "SecurityPkg.dec" itself (currently it only allows fixed-at-build or patchable, not dynamic). Do you want me to file a BZ in the TianoCore tracker for this, and assign it to you? If you don't have time for writing the patch, I'm glad to do it too, but then the review could be slower; both other OvmfPkg co-maintainers are busy with other things.) Thanks! Laszlo
On 9/29/17 4:58 PM, Laszlo Ersek wrote: ...
The expansion ROMs (containing UEFI drivers) of emulated PCI devices, and the same of assigned physical PCI devices, constitute another channel through which code enters the guest from the outside (i.e., from the Cloud Provider). The ROM BARs from which the guest firmware reads the UEFI binaries are not guest RAM, they are MMIO. (For execution, the drivers are copied into encrypted guest RAM.)
If the guest has Secure Boot enabled, then the oproms are verified[*] (and not launched if verification fails), but this is slightly different from what I understand under audit-by-GO. It means the GO wouldn't get a measurement of the oproms for one-by-one clearing, when about to green-light a guest startup. Instead the GO would ensure that Secure Boot be enabled with the right certificates (and/or executable hashes) enrolled off the bat, and then implicitly trust all oprom drivers accepted by those certs / hashes. It's another layer of indirection.
This is likely nothing new qualitatively, but "the devil is in the details", so I thought it was worth raising.
[*] For edk2 / OvmfPkg specifics, I'll mention
gEfiSecurityPkgTokenSpaceGuid.PcdOptionRomImageVerificationPolicy
The SecurityPkg default is 0x04 ("Deny execution when there is security violation"). However, OVMF sets it to 0x00 ("Always trust the image"). Please see the following commit for the reasons:
https://github.com/tianocore/edk2/commit/1fea9ddb4e3fd
Brijesh, for SEV guests, we likely want to flip this PCD to 0x04, in the AmdSevInitialize() function, in "OvmfPkg/PlatformPei/AmdSev.c". For that we'll also have to change the PCD from fixed-at-build to dynamic, but that in turn will require a change to "SecurityPkg.dec" itself (currently it only allows fixed-at-build or patchable, not dynamic). Do you want me to file a BZ in the TianoCore tracker for this, and assign it to you? If you don't have time for writing the patch, I'm glad to do it too, but then the review could be slower; both other OvmfPkg co-maintainers are busy with other things.)
Very good point Laszlo. Please submit the BZ and assign it me - thank you. we will take a look at implementing the required support. -Brijesh
On 10/01/17 02:09, Brijesh Singh wrote:
On 9/29/17 4:58 PM, Laszlo Ersek wrote: ...
The expansion ROMs (containing UEFI drivers) of emulated PCI devices, and the same of assigned physical PCI devices, constitute another channel through which code enters the guest from the outside (i.e., from the Cloud Provider). The ROM BARs from which the guest firmware reads the UEFI binaries are not guest RAM, they are MMIO. (For execution, the drivers are copied into encrypted guest RAM.)
If the guest has Secure Boot enabled, then the oproms are verified[*] (and not launched if verification fails), but this is slightly different from what I understand under audit-by-GO. It means the GO wouldn't get a measurement of the oproms for one-by-one clearing, when about to green-light a guest startup. Instead the GO would ensure that Secure Boot be enabled with the right certificates (and/or executable hashes) enrolled off the bat, and then implicitly trust all oprom drivers accepted by those certs / hashes. It's another layer of indirection.
This is likely nothing new qualitatively, but "the devil is in the details", so I thought it was worth raising.
[*] For edk2 / OvmfPkg specifics, I'll mention
gEfiSecurityPkgTokenSpaceGuid.PcdOptionRomImageVerificationPolicy
The SecurityPkg default is 0x04 ("Deny execution when there is security violation"). However, OVMF sets it to 0x00 ("Always trust the image"). Please see the following commit for the reasons:
https://github.com/tianocore/edk2/commit/1fea9ddb4e3fd
Brijesh, for SEV guests, we likely want to flip this PCD to 0x04, in the AmdSevInitialize() function, in "OvmfPkg/PlatformPei/AmdSev.c". For that we'll also have to change the PCD from fixed-at-build to dynamic, but that in turn will require a change to "SecurityPkg.dec" itself (currently it only allows fixed-at-build or patchable, not dynamic). Do you want me to file a BZ in the TianoCore tracker for this, and assign it to you? If you don't have time for writing the patch, I'm glad to do it too, but then the review could be slower; both other OvmfPkg co-maintainers are busy with other things.)
Very good point Laszlo. Please submit the BZ and assign it me - thank you. we will take a look at implementing the required support.
I've filed: https://bugzilla.tianocore.org/show_bug.cgi?id=728 Another thought: once we delegate part of the protection that SEV provides to the Secure Boot implementation in the guest, the pflash chip that stores the UEFI variables (i.e., certificates for SB as well) becomes a target of attacks not just from within the guest, but also from the host. This is a big problem because the pflash chip is MMIO, nor RAM. So it is not encrypted, and the Cloud Provider can tamper with it at will -- even if the variable store file that backs the pflash chip (with the certificates pre-enrolled) comes from the Guest Owner initially, and/or passes the GO's measurement check. Put differently, even if we set PcdOptionRomImageVerificationPolicy to 0x04, it doesn't help if the CP can poke any UEFI executable hash or self-signed certificate into the varstore, which in turn will "verify" the CP's malicious oprom driver. The varstore pflash chip has never been considered to be under attack from the *host* side; all the guest-SMM work was focused on protecting pflash (and guest firmware in general) from the *guest* OS (and 3rd party *guest* UEFI apps). Theoretically, I see three ways to mitigate this: (1) Eliminate pflash from OVMF altogether. This would be a huge project, affecting OVMF, QEMU and libvirt. It would also throw the current in-guest protections provided by SB out the window. (2) Keep pflash as-is, but do not off-load any SEV protections to guest-side Secure Boot. This means that the guest firmware must be comprehensively restricted to load binaries only from media that the GO measured and the CP cannot tamper with. No option ROMs, no PXE boot, no plaintext ISOs, no plaintext hard disk images. (And booting from encrypted ISOs / hard disk images would require a LUKS driver for OVMF.) (3) Implement SEV encryption for pflash. A pflash chip can be in one of two modes: (a) it reads and executes as ROM, or (b) it behaves like a programmable (r/w) device with MMIO registers. Switching between both modes is explicit (see "OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlash.c"); perhaps some SEV controls could be hooked in there. Thanks Laszlo
On 10/01/17 11:17, Laszlo Ersek wrote:
(3) Implement SEV encryption for pflash. A pflash chip can be in one of two modes: (a) it reads and executes as ROM, or (b) it behaves like a programmable (r/w) device with MMIO registers. Switching between both modes is explicit (see "OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlash.c"); perhaps some SEV controls could be hooked in there.
Hmmm wait a second, we *already* execute code from pflash (namely SEC code from OVMF_CODE.fd), and instruction fetch always entails decryption. Furthermore, in SEC, we decompress PEI and DXE FVs from pflash (OVMF_CODE.fd) to RAM -- the decompression code runs in long mode, plus you modified the OVMF/X64 reset vector to set the C-bit in all PTEs, covering the low 4GB of guest RAM. This tells me that at least read and execute from pflash work with SEV decryption already. Is this expected? However, in PlatformPei, we don't report the varstore pflash range via any resource descriptor HOB (the varstore chip is at [0xffc00000, 0xFFC84000) in the 4MB build). Consequently, AmdSevDxe clears the C-bit under the "NonExistent" branch: AmdSevDxe:SetMemoryEncDec: Cr3Base=0x0 Physical=0xFEF00000 Length=0x1100000 Mode=Decrypt CacheFlush=0 Subsequently, "OvmfPkg/QemuFlashFvbServicesRuntimeDxe/FwBlockService.c" adds the pflash range as EFI_MEMORY_UC | EFI_MEMORY_RUNTIME. The C-bit remains clear, AIUI. OK, I'm totally confused now. Looks like SEV decryption already works for pflash (from SEC's example), and transparently at that (???), but we fail to use it in QemuFlashFvbServicesRuntimeDxe? Laszlo
Hi Laszlo, On 10/01/2017 04:56 AM, Laszlo Ersek wrote:
On 10/01/17 11:17, Laszlo Ersek wrote:
(3) Implement SEV encryption for pflash. A pflash chip can be in one of two modes: (a) it reads and executes as ROM, or (b) it behaves like a programmable (r/w) device with MMIO registers. Switching between both modes is explicit (see "OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlash.c"); perhaps some SEV controls could be hooked in there.
Hmmm wait a second, we *already* execute code from pflash (namely SEC code from OVMF_CODE.fd), and instruction fetch always entails decryption.
Furthermore, in SEC, we decompress PEI and DXE FVs from pflash (OVMF_CODE.fd) to RAM -- the decompression code runs in long mode, plus you modified the OVMF/X64 reset vector to set the C-bit in all PTEs, covering the low 4GB of guest RAM.
This tells me that at least read and execute from pflash work with SEV decryption already. Is this expected?
Yes, this is expected. During the machine_init, qemu creates two pflash units. The pflash unit#0 contains OVMF_CODE.fd and unit #1 points to OVMF_VARS.fd. As part of machine creation, we encrypt the unit#0 using KVM_SEV_LAUNCH_UPDATE command. The unit#0 is mapped at the x86 reset vector. On vm_start, guest fetches the code from the unit#0. Since the code was already encrypted hence guest was able to read and execute the initial bootstrap code just fine. When OVMF is build with OvmfPkgX64.dsc, SetCr3ForPageTables64 builds the initial page table. When SEV is active, we add the C-bit pte mask [1] -- it will ensures that when decompression happen then data get written as encrypted in the guest RAM. [1] https://github.com/tianocore/edk2/commit/e60af8a1ebb15bfcbf2ecc4afb6cf35084c... When OVMF is build with OvmfPkgIa32X64.dsc, the SEC phase runs in 32-bit mode hence SEV hardware forces the C-bit to 1. This will ensure that decompression routines writes the data as encrypted.
However, in PlatformPei, we don't report the varstore pflash range via any resource descriptor HOB (the varstore chip is at [0xffc00000, 0xFFC84000) in the 4MB build). Consequently, AmdSevDxe clears the C-bit under the "NonExistent" branch:
AmdSevDxe:SetMemoryEncDec: Cr3Base=0x0 Physical=0xFEF00000 Length=0x1100000 Mode=Decrypt CacheFlush=0
Subsequently, "OvmfPkg/QemuFlashFvbServicesRuntimeDxe/FwBlockService.c" adds the pflash range as EFI_MEMORY_UC | EFI_MEMORY_RUNTIME. The C-bit remains clear, AIUI.
OK, I'm totally confused now. Looks like SEV decryption already works for pflash (from SEC's example), and transparently at that (???), but we fail to use it in QemuFlashFvbServicesRuntimeDxe?
AmdSevDxe clears the C-bit of pflash range in PlatformPei because QemuFlashFvbServicesRuntimeDxe switches the pflash out from ROMD mode; i.e it become MMIO. The MMIO regions must be mapped as unencrypted. SEV hardware engine uses a random encryption key on each run. Since OVMF_VAR.fd is used for storing the UEFI persistent variable hence we will not be able to store the data encrypted using the SEV engine. Any persistent storage should be protected using a disk encryption technologies (like luks, dm-crypt etc). Regarding optionROM comments, I will propose something on edk-devel list. I am thinking at least we should start by adding some kind of bug_on or warning when a SEV guest BIOS is attempting to copy the option ROM. I am hoping that once we have secure boot working then we should be able to handle the signed option ROM, grub.efi or kernel.efi etc. -Brijesh
participants (8)
-
Brijesh Singh -
Daniel P. Berrange -
Dr. David Alan Gilbert -
Erik Skultety -
Laszlo Ersek -
Michael S. Tsirkin -
Relph, Richard -
Richard Relph