Hi Daniel, QEMU/KVM work is in process of completion and would be published soon for the community approval(somewhere in March). We are working closely with development team of KVM/QEMU. Meanwhile, we wanted to start the process in Libvirt community. We will submit the work status of QEMU/KVM as they are available. We are new to the Libvirt community and wanted to know more about the Libvirt design approval process, and wanted to kick start our work Question we have are 1. What is the blueprint or design template for a feature approval?. 2. Procedure and requirements for design and code approval. 3. we wanted to use Libvirt MKTME in Openstack, what are the requirements? Yes you are right, SEV and MKTME are quite different , we were just mentioning the similarities of the features. We will definitely point out if they are any overlapping in the development of these two. Thanks Karim. -----Original Message----- From: Daniel P. Berrangé [mailto:berrange@redhat.com] Sent: Monday, March 4, 2019 2:33 AM To: Carvalho, Larkins L <larkins.l.carvalho@intel.com> Cc: libvir-list@redhat.com; Mohammed, Karimullah <karimullah.mohammed@intel.com> Subject: Re: [libvirt] New Feature: Intel MKTME Support On Thu, Feb 28, 2019 at 11:16:30PM +0000, Carvalho, Larkins L wrote:

Hello Team,

Greetings. We want to add Intel MKTME support to the Libvirt. Intel MKTME is a capability to encrypt entirety of physical memory of a system similar to AMD SEV.

Please let us know what are the expectations from us to initiate the design and development of the feature.

Libvirt is likely dependant on QEMU / KVM to implement the low level parts of this feature. So what is the status of QEMU / KVM work in this area ? If it already exists, can you outline how it is used. I get the feeling the impl is quite different from AMD SEV, but if there's any scope to use similar/overlapping libvirt design in libvirt that is highly desirable. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Hi Daniel, Thank you for answering our questions. We will soon send our design documentation for a review/discussion for MKTME enablement. This is not a complex feature , but in any case we wanted to start off with a design review , so that we get approved forehand for what we will be implementing. I would like to take liberty in asking you question related to Libvirt, I did ask this question in IRC channel did not get any responses. Can Libvirt directly make an kernel system call? i.e for a XML request if we have to make a kernel syscall, can we directly make kernel syscall in Libvirt or do we have to go through QEMU to process the request. We would like to know the norm of calling kernel system calls in Libvirt. Thanks karim -----Original Message----- From: Daniel P. Berrangé [mailto:berrange@redhat.com] Sent: Monday, March 4, 2019 1:47 PM To: Mohammed, Karimullah <karimullah.mohammed@intel.com> Cc: Carvalho, Larkins L <larkins.l.carvalho@intel.com>; libvir-list@redhat.com Subject: Re: [libvirt] New Feature: Intel MKTME Support On Mon, Mar 04, 2019 at 07:54:30PM +0000, Mohammed, Karimullah wrote:

Hi Daniel, QEMU/KVM work is in process of completion and would be published soon for the community approval(somewhere in March). We are working closely with development team of KVM/QEMU. Meanwhile, we wanted to start the process in Libvirt community. We will submit the work status of QEMU/KVM as they are available.

We are new to the Libvirt community and wanted to know more about the Libvirt design approval process, and wanted to kick start our work

Question we have are 1. What is the blueprint or design template for a feature approval?. 2. Procedure and requirements for design and code approval.

There's no formal process mandated for libvirt. Either people just send code patches directly, or for more complex features like this, start off with a mailing list discussion about design decisions before writing patches. Libvirt releases once a month so there's no critical deadlines you need to rush to meet from upstream POV - they'll merge in whatever release is next once the code has passed review. For complex code it is common to post several versions adapting to review comments. In terms of merging code, if it depends on QEMU, then we require that the QEMU impl is merged before the libvirt impl is merged. We'd encourage you to develop & publish libvirt patches before QEMU work is finalized. This helps demonstrate that the QEMU impl is a good match for what libvirt needs.

3. we wanted to use Libvirt MKTME in Openstack, what are the requirements?

This is probably more a question for the OpenStack community to answer. The goal with Libvirt is to expose the feature in XML and/or API in a way that is suitable for any mgmt application. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Hi Daniel, MKTME supports encryption of memory(NVRAM) for Virtual Machines(hardware based encryption). This features uses Linux kernel key ring services, i.e. Operations like, allocation and clearing of secret/keys. These keys are used in encryption of memory in Virtual machines. So MKTME provided encryption of entire RAM of a VM, allocated to it, thereby supporting VM isolation feature. So to implement this functionality in openstack 1. Nova executes host capability command, to identify if the hardware support for MKTME (openstack xml host_capabilities command request -->> libvirt ->> QEMU)-- qemu monitoring commands 2. Once the hardware is identified and if user configures mktme policy to launch a VM in openstack, Nova a. Sends a new xml command request to libvirt, then libvirt makes a syscall to Linux kernel key ring services to get/retrieve a key/key-handle for this VM ( we are not sure at this point whether to make this syscall directly in libvirt or through QEMU) b. Once the key is retrieved , Nova compute executes a VM launch xml command request to libvirt with a new argument called mktme- keyhandle , which will send a command request to QEMU to launch the VM( We are in process of supporting this functionality in QEMU for VM launch operation, with new mktme-key argument) We are not sure , where to make this(2a) kernel system calls at present and looking for suggestions. Thanks karim -----Original Message----- From: Daniel P. Berrangé [mailto:berrange@redhat.com] Sent: Tuesday, March 5, 2019 2:15 AM To: Mohammed, Karimullah <karimullah.mohammed@intel.com> Cc: Carvalho, Larkins L <larkins.l.carvalho@intel.com>; libvir-list@redhat.com Subject: Re: [libvirt] New Feature: Intel MKTME Support On Mon, Mar 04, 2019 at 10:44:12PM +0000, Mohammed, Karimullah wrote:

Hi Daniel,

Thank you for answering our questions. We will soon send our design documentation for a review/discussion for MKTME enablement. This is not a complex feature , but in any case we wanted to start off with a design review , so that we get approved forehand for what we will be implementing.

I would like to take liberty in asking you question related to Libvirt, I did ask this question in IRC channel did not get any responses.

Can Libvirt directly make an kernel system call? i.e for a XML request if we have to make a kernel syscall, can we directly make kernel syscall in Libvirt or do we have to go through QEMU to process the request. We would like to know the norm of calling kernel system calls in Libvirt.

It is hard to give a general answer to that without understanding the context of the system call in question. Libvirt can certainly make arbitrary system calls as it needs. If the system call is discovering information that has an impact on QEMU functionality though, it may be better to query it via QEMU. If you can provide more detail & usage context we can give a more useful answer. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Tue, Mar 05, 2019 at 05:35:09PM +0000, Daniel P. Berrangé wrote:

On Tue, Mar 05, 2019 at 05:23:04PM +0000, Mohammed, Karimullah wrote:

...

Hi Daniel, MKTME supports encryption of memory(NVRAM) for Virtual Machines(hardware based encryption). This features uses Linux kernel key ring services, i.e. Operations like, allocation and clearing of secret/keys. These keys are used in encryption of memory in Virtual machines. So MKTME provided encryption of entire RAM of a VM, allocated to it, thereby supporting VM isolation feature.

So to implement this functionality in openstack

1. Nova executes host capability command, to identify if the hardware support for MKTME (openstack xml host_capabilities command request -->> libvirt ->> QEMU)-- qemu monitoring commands 2. Once the hardware is identified and if user configures mktme policy to launch a VM in openstack, Nova a. Sends a new xml command request to libvirt, then libvirt makes a syscall to Linux kernel key ring services to get/retrieve a key/key-handle for this VM ( we are not sure at this point whether to make this syscall directly in libvirt or through QEMU)

What will openstack do with the key / key-handle it gets back from libvirt ?

Same question here.

Why does it need to allocate one before starting the VMs, as opposed to letting QEMU or libvirt allocate it during startup ?

By allocating it separately from the VM start request it opens the possibility for leaking keys, if VM startup fails and the mgmt app doesn't release the now unused key.

I would expect this key/handle work similarly as it does with SEV, we (libvirt) treat everything as a blob since the session key is encrypted by a transport key shipped along with an integrity key which were derived by a master secret both parties know. My question is whether you have a draft of this MKTME spec that we could have a look at to give us more technical insight and therefore help us to make better design decisions.

...

b. Once the key is retrieved , Nova compute executes a VM launch xml command request to libvirt with a new argument called mktme- keyhandle , which will send a command request to QEMU to launch the VM( We are in process of supporting this functionality in QEMU for VM launch operation, with new mktme-key argument)

As Dan asked above, this really depends on why does openstack need to interact with the key and whether the key handle can be computed during the launch phase. For example, in SEV's case we pass the VM owner's certificate to the SEV firmware as part of the VM configuration and the handshake and a measurement verification both happen after we initialized QEMU and if necessary (measurement purposes) we start the VM in the paused state so that commands can be passed to QEMU handling all the interactions with SEV in the kernel instead of us. Thanks, Erik

Hi Erik and Daniel, Earlier we were considering a different method of accessing Linux kernel ring services to get a key-handle i.e through daemon process in openstack. But later on we decided to do it through the Libvirt as Daemon processes is not feasible in Openstack. Hence we had separate calls for getting key-handle and using that key launching VM through Libvirt. However we are open for, the idea of getting key-handle during startup of Libvirt. Looks like getting key-handle in QEMU is not possible because of some limitations in the code it seems(we will give you more info on this one, once we have accurate information from our QEMU team). We want to understand following things in virtualization. 1. What do you mean by " letting QEMU or libvirt to allocate key-handle during startup ?. Is this means, before launching a VM get key-handle in either in Libvirt/QEMU?.. if so next tpoint 2. If not in QEMU, can we make Linux syscalls to get key-handle before launching a command to QEMU during VM launch? In Libvirt? like . Nova compute during launch of an VM , sends mktme policy in the same xml file to Libvirt. . Libvirt makes kernel syscall to get key-handle and then sends QEMU command to launch the VM with addition of key-handle parameter Here is the brief presentation of, how we plan to execute mtkme end to end in openstack, so that we are on same page.. 1. Cloud service provider(CSP) launches a VM instance using mktme policy in Image meta-data Mktme-policy { Mktme-key-id = "Mname1" (String), Mktme-key-type = cpu or user (CPU = hardware generated key, user = user given key) Mktme-key-url = https://xx.xx.xx ( URL to fetch the key) } 2. After all checks , Nova compute will get a command to launch a VM, if mtkme-policy to be found. If mktme-key-type is user, it gets the key from the url specified in mktme-key-url and that key is called mktme-key. If mktme-key-type is cpu no action and mktme-key will be null. 3. Assuming key-handle and VM launch are separate calls. Nova compute execute a new libvirt command to get the key-handle given the arguments { Mktme-key-id, Mktme-key-type, Mktme-key(if user type key). }. a. Libvirt make a Linux kernel ring service syscall request_key( with above parameters), request_key syscall return a key-handle if a key exist with mktme-key-id or it will create a new key-handle and returns it. This is true for both user and cpu type keys. (Now this command can also be extended/execute in QEMU) b. Nova gets the key-handle in return and launched the VM instance using this additional key-handle argument to Libvirt again. 4. Assuming key-handle is done in Libvirt( if I'm not mistaken, this what you were proposing). Nova compute execute VM launch libvirt call using mktme additional parameters { Mktme-key-id, Mktme-key-type, Mktme-key(if user type key, ). } a. Libvirt upon receiving this call , execute request_key kernel syscall using the above parameters and gets the key-handle and thereby launches the VM using the usual QEMU command with addition parameter of mktme key-handle. And again this whole process can also get executed in QEMU, but we have some limitation I guess at this point). AMS SEV and MKTME are not quite similar, but we are open to execute key-handle get/set operations in libvirt at this point. We have seen lot of pros in executing the things the way you were proposing. Like, the key leakage, number of calls to Libvirt will be minimized and less Nova code and modification. We really appreciate your suggestions and advices , hope the above explanation helps understand our design. We are open for any changes , that gives us a good design in libvirt. Here is the link to mktme spec : https://software.intel.com/sites/default/files/managed/a5/16/Multi-Key-Total... Thanks again Karim. -----Original Message----- From: Erik Skultety [mailto:eskultet@redhat.com] Sent: Wednesday, March 6, 2019 12:08 AM To: Daniel P. Berrangé <berrange@redhat.com> Cc: Mohammed, Karimullah <karimullah.mohammed@intel.com>; libvir-list@redhat.com; Carvalho, Larkins L <larkins.l.carvalho@intel.com> Subject: Re: [libvirt] New Feature: Intel MKTME Support On Tue, Mar 05, 2019 at 05:35:09PM +0000, Daniel P. Berrangé wrote:

On Tue, Mar 05, 2019 at 05:23:04PM +0000, Mohammed, Karimullah wrote:

...

Hi Daniel, MKTME supports encryption of memory(NVRAM) for Virtual Machines(hardware based encryption). This features uses Linux kernel key ring services, i.e. Operations like, allocation and clearing of secret/keys. These keys are used in encryption of memory in Virtual machines. So MKTME provided encryption of entire RAM of a VM, allocated to it, thereby supporting VM isolation feature.

So to implement this functionality in openstack

1. Nova executes host capability command, to identify if the hardware support for MKTME (openstack xml host_capabilities command request -->> libvirt ->> QEMU)-- qemu monitoring commands 2. Once the hardware is identified and if user configures mktme policy to launch a VM in openstack, Nova a. Sends a new xml command request to libvirt, then libvirt makes a syscall to Linux kernel key ring services to get/retrieve a key/key-handle for this VM ( we are not sure at this point whether to make this syscall directly in libvirt or through QEMU)

What will openstack do with the key / key-handle it gets back from libvirt ?

Same question here.

Why does it need to allocate one before starting the VMs, as opposed to letting QEMU or libvirt allocate it during startup ?

By allocating it separately from the VM start request it opens the possibility for leaking keys, if VM startup fails and the mgmt app doesn't release the now unused key.

I would expect this key/handle work similarly as it does with SEV, we (libvirt) treat everything as a blob since the session key is encrypted by a transport key shipped along with an integrity key which were derived by a master secret both parties know. My question is whether you have a draft of this MKTME spec that we could have a look at to give us more technical insight and therefore help us to make better design decisions.

...

b. Once the key is retrieved , Nova compute executes a VM launch xml command request to libvirt with a new argument called mktme- keyhandle , which will send a command request to QEMU to launch the VM( We are in process of supporting this functionality in QEMU for VM launch operation, with new mktme-key argument)

As Dan asked above, this really depends on why does openstack need to interact with the key and whether the key handle can be computed during the launch phase. For example, in SEV's case we pass the VM owner's certificate to the SEV firmware as part of the VM configuration and the handshake and a measurement verification both happen after we initialized QEMU and if necessary (measurement purposes) we start the VM in the paused state so that commands can be passed to QEMU handling all the interactions with SEV in the kernel instead of us. Thanks, Erik

Hi Daniel, Thank you for your valuable suggestion and we started looking into Libvirt code in that perspective and will soon send a low level design for our implementation of keyctl calls in Libvirt. We will put together QEMU monitoring commands as well as vm launch, using mktme keys in one design doc and will share with you all. Thanks Karim P. S : Was not able to set the wrapper in outlook for some reason, still figuring out the issue. Meanwhile adding line breaks , hope this is good. -----Original Message----- From: Daniel P. Berrangé [mailto:berrange@redhat.com] Sent: Wednesday, March 6, 2019 3:36 AM To: Mohammed, Karimullah <karimullah.mohammed@intel.com> Cc: Erik Skultety <eskultet@redhat.com>; libvir-list@redhat.com; Carvalho, Larkins L <larkins.l.carvalho@intel.com>; Huang, Kai <kai.huang@intel.com> Subject: Re: [libvirt] New Feature: Intel MKTME Support BTW, if possible, could you configure your email client to line wrap text at 72 columns. It is currently sending messages with no line breaks except at end of paragraph which means I have to word wrap your quoted text myself when replying. On Wed, Mar 06, 2019 at 10:27:59AM +0000, Mohammed, Karimullah wrote:

We want to understand following things in virtualization.

1. What do you mean by " letting QEMU or libvirt to allocate key-handle during startup ?. Is this means, before launching a VM get key-handle in either in Libvirt/QEMU?.. if so next tpoint

I mean do it during the virDomainCreate API call, which is the call a mgmt app already makes to start a VM.

2. If not in QEMU, can we make Linux syscalls to get key-handle before launching a command to QEMU during VM launch? In Libvirt? like

. Nova compute during launch of an VM , sends mktme policy in the same xml file to Libvirt. . Libvirt makes kernel syscall to get key-handle and then sends QEMU command to launch the VM with addition of key-handle parameter

Here is the brief presentation of, how we plan to execute mtkme end to end in openstack, so that we are on same page.. 1. Cloud service provider(CSP) launches a VM instance using mktme policy in Image meta-data Mktme-policy { Mktme-key-id = "Mname1" (String), Mktme-key-type = cpu or user (CPU = hardware generated key, user = user given key) Mktme-key-url = https://xx.xx.xx ( URL to fetch the key) } 2. After all checks , Nova compute will get a command to launch a VM, if mtkme-policy to be found. If mktme-key-type is user, it gets the key from the url specified in mktme-key-url and that key is called mktme-key. If mktme-key-type is cpu no action and mktme-key will be null.

3. Assuming key-handle and VM launch are separate calls. Nova compute execute a new libvirt command to get the key-handle given the arguments { Mktme-key-id, Mktme-key-type, Mktme-key(if user type key). }. a. Libvirt make a Linux kernel ring service syscall request_key( with above parameters), request_key syscall return a key-handle if a key exist with mktme-key-id or it will create a new key-handle and returns it. This is true for both user and cpu type keys. (Now this command can also be extended/execute in QEMU) b. Nova gets the key-handle in return and launched the VM instance using this additional key-handle argument to Libvirt again.

Nova is requesting a value from libvirt and then feeding it straight back to libvirt. This is the bit that looks pointless to me. It is appearing to add an extra API call for no benefit. The only reason to have a seperate API call is if Nova needs to do some kind of computational work with the key it gets in step (a) before it then gives it back to libvirt in step (b).

4. Assuming key-handle is done in Libvirt( if I'm not mistaken, this what you were proposing). Nova compute execute VM launch libvirt call using mktme additional parameters { Mktme-key-id, Mktme-key-type, Mktme-key (if user type key, ). } a. Libvirt upon receiving this call , execute request_key kernel syscall using the above parameters and gets the key-handle and thereby launches the VM using the usual QEMU command with addition parameter of mktme key-handle. And again this whole process can also get executed in QEMU, but we have some limitation I guess at this point).

I think libvirt would still talk to the kernel to get the key. Since the keys are a finite resource, we don't want to give QEMU the permission to request keys itself as that opens a denial of service possibility. QEMU should only be able to use keys it has been given by libvirt, which means libivrt must request them from the kernel. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

Hi Daniel, Just FYI we figured out how to generate domain and qemucaps xml files. For qemucaps xml files , we ran the libvirtd process and located the xml files in /var/cache/libvirtd/qemu/capabilities directory. And was able to generate domaincaps using virsh command under tools directory. We are using our locally modified qemu binary for these procedures. Thanks karim -----Original Message----- From: Mohammed, Karimullah Sent: Friday, April 19, 2019 1:09 AM To: Daniel P. Berrangé <berrange@redhat.com> Cc: Carvalho, Larkins L <larkins.l.carvalho@intel.com>; libvir-list@redhat.com Subject: RE: [libvirt] New Feature: Intel MKTME Support Hi Daniel, After adding code for this feature, when we run "make check" as expected we are failing at two test cases (domaincapstest, qemucapabilitiestest). So in order to test with right data we figured out that we need to add new .xml files with mktme qemu info in qemucapabilitiesdata/ and domaincapsschemadata/ directories and .replies file. We have internal qemu binary with mktme queries enabled. We figured out how to generate a new .replies file with our internal qemu binary using qemucapsprobe executable. Could you please help us in understanding the libvirt test directory qemu and caps xml files? We have the following questions. 1. How to do we generate .xml files with new mktme data in both qemucapabilitiesdata/ and domaincapsschemadata/ directories?. Or for time being , can we add mktme info to existing .xml files lets sat caps_4.0.0.x86-64.xml and qemu_4.0.0.x86-64.xml?. 2. Do we have to pass all these test cases (make check) before pushing a patch? Just for the review. We want to get preliminary review feedback from the libvirt community for our feature. 3. we are planning to generate the .replies file using our internal qemu binary to make sure we pass unit and functional test cases with "make check", because for this feature as I mentioned earlier QEMU is not available until mid of next month, is this ok? Thanks karim -----Original Message----- From: Daniel P. Berrangé [mailto:berrange@redhat.com] Sent: Tuesday, March 5, 2019 9:35 AM To: Mohammed, Karimullah <karimullah.mohammed@intel.com> Cc: Carvalho, Larkins L <larkins.l.carvalho@intel.com>; libvir-list@redhat.com Subject: Re: [libvirt] New Feature: Intel MKTME Support On Tue, Mar 05, 2019 at 05:23:04PM +0000, Mohammed, Karimullah wrote:

Hi Daniel, MKTME supports encryption of memory(NVRAM) for Virtual Machines(hardware based encryption). This features uses Linux kernel key ring services, i.e. Operations like, allocation and clearing of secret/keys. These keys are used in encryption of memory in Virtual machines. So MKTME provided encryption of entire RAM of a VM, allocated to it, thereby supporting VM isolation feature.

So to implement this functionality in openstack

1. Nova executes host capability command, to identify if the hardware support for MKTME (openstack xml host_capabilities command request -->> libvirt ->> QEMU)-- qemu monitoring commands 2. Once the hardware is identified and if user configures mktme policy to launch a VM in openstack, Nova a. Sends a new xml command request to libvirt, then libvirt makes a syscall to Linux kernel key ring services to get/retrieve a key/key-handle for this VM ( we are not sure at this point whether to make this syscall directly in libvirt or through QEMU)

What will openstack do with the key / key-handle it gets back from libvirt ? Why does it need to allocate one before starting the VMs, as opposed to letting QEMU or libvirt allocate it during startup ? By allocating it separately from the VM start request it opens the possibility for leaking keys, if VM startup fails and the mgmt app doesn't release the now unused key.

b. Once the key is retrieved , Nova compute executes a VM launch xml command request to libvirt with a new argument called mktme- keyhandle , which will send a command request to QEMU to launch the VM( We are in process of supporting this functionality in QEMU for VM launch operation, with new mktme-key argument)

We are not sure , where to make this(2a) kernel system calls at present and looking for suggestions.

Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|