Daniel, thanks for the review. ----- "Daniel P. Berrange" <berrange(a)redhat.com&gt; wrote: The format of encrypted data is conceptually separate from the format of the image: for example, a future LVM crypt implementation is planned that will support on-line encryption key change (re-encrypting the data without needing to stop its users). This can be naturally used on partitions, but it can also be used on a ("loopback-mounted") file stored on another filesystem - which is useful because this gives us on-line encryption key change capability for both partitions and files without creating two separate implementations of the functionality. (The same loopback setup could be done with LUKS today, although it is rather pointless.) So the fact that qemu treats the unencrypted file using the "raw" format does not imply that no encryption is used, or that any particular encryption implementation is used. For these reasons I think the encryption format should be explicitly specified. (The same information could be represented in the <driver> element, but there is currently no equivalent generic functionality for the <volume> element. It is also better to keep this information "within the <encryption> node", because this allows the "dumb" client discussed below to simply store the <encryption> node for each volume, and attach it inside <domain> descriptions, without any further understanding of the encryption format.) OK, I'll change that. Exactly. The expected content of the <encryption> element is "format"-specific. Not quite: the main case of a "dumb" client would be a large-scale virtualization management software that contains a primary store of encryption information, and gives each node access only to those keys that are currently necessary by the node to run its domains; the fact that each node has access to only a limited set of keys prevents an attacker that compromises a single node from reading disk images of all domains managed in the entire site, even if the disk image storage (e.g. unencrypted NFS) does not allow managing access by each node separately. Such a client must be able to transfer the actual secrets, not only identifiers, to libvirt. (The idea of a "dumb" client, that does not know the specifics of the format, is an additional feature on top, but one that implies that the client does send the actual secrets.) Storage of secrets in a separate keystore is more important for "local" libvirt deployments, where libvirt manages the primary, long-term, store of the secrets. Another option here is to let the client store the secrets in gnome-keyring, and transfer them to libvirt only when starting a domain (especially when there are no persistent domains). That doesn't affect the design in any way, though. That is unreliable with current implementation, because a pool refresh creates all information about volumes anew by reading the volume files; therefore, after any pool refresh libvirt "forgets" the secrets directly associated with a volume (not secrets associated with an use of a volume in a domain). If client A creates a volume, client B can refresh the pool before client A is able to read the automatically generated secrets. One of the reasons the patch is clearing the information immediately is to make sure a {virStorageVolCreateXML; virStorageVolGetXMLDesc) always fails and no client is written depending on this racy operation sequence. There is another, more important, reason why the node should never return encryption data to anyone who can connect to it. Consider the above-described situation of a large-scale virtualization deployment that uses volume image encryption to restrict access of nodes to volume data: If domain migration is supported, the nodes (all of them, or at least nodes within some groups) must be able to connect "read-write" to other nodes. Yet, if the volume image encryption is to restrict access to volume data, such connections must not allow extracting encryption secrets, or any compromised node could simply ask the other nodes for their secrets and read the data. Thus, even a read-write connection should not allow any client to read secrets. I did not realize this when I started writing the original patches; I'll modify them not to allow returning the secrets using virDomainGetXMLDesc(), which is currently possible. The only potential exception is volume creation with secret auto-generation (not implemented in these patches), and in this case it is important to ensure that only the client that initiated the volume creation, not other clients, is can get the generated secrets. The most natural implementation would be a new operation (virStorageVolCreateXMLFromAndReturn???) that generates the secrets, creates the volume, and returns the secrets in a single step. In this scenario virStorageVolDumpXML does not need to return the secrets. As argued below, that might not be necessary, depending on the intended scope of libvirt. I wouldn't necessarily be that strict - if a centralized key management server is used by libvirt, it might still make sense for users to create one-off virtual machines that are encrypted, but not managed by the centralized infrastructure. We might not need all of these. There are three main use cases for storing keys outside of XML: 1) This is a "small-scale deployment", where libvirt is the primary key store, and anyone that is able to connect to libvirtd is implicitly allowed to use the secrets. In this case the keystore can be managed completely automatically - creating a volume, or perhaps using it for the first time, implies storing a secret, and deleting a volume implies deleting a secret. Secrets are identified using a volume-unique object (is that a path, or an UUID?), which is completely transparent to the client. As long as the volume is "known" to libvirt, the client does not even need to specify any <encryption> element when creating a domain. (1a, a "small-scale deployment" where there are multiple clients that should be protected against each other, and libvirt is the primary key store, does not make sense unless a client account system is added to libvirt. If libvirt can not distinguish between different client accounts, it can only restrict access to secrets by insisting that the client specifies a keyuuid - but the client will need to store the keyuuid values and protect them exactly as if they were the "real" secrets; the client can just as easily store the "real" secrets and skip using keyuuids altogether.) 2) The secrets are primarily stored outside libvirt (to restrict node access to image data), and libvirt stores only secrets for currently defined persistent domains (to support domain autostart). In this case the keystore can be managed completely automatically - persistent domain definition implies storing a secret, deleting the domain implies deleting a secret. Secrets are identified using a (domain, volume) pair, and this identifier is never exposed to the client. 3) An external key store (such as a KMIP server) is used. In this case the management of access rights, listing and deleting secrets, would be performed by interacting with the external key store directly. Secrets are identified by using the external key store identifiers, and the client and libvirtd send these identifiers to each other. The above seems to indicate the only operations libvirt needs from a keystore are: - generate a UUID (in a keystore-specific format) - store a secret using a specified UUID - get secret associated with an UUID - delete a secret associated with an UUID. The first two operations could perhaps be combined, depending on the specifics of the external key store (e.g. the client might want to allocate an UUID so that the client, not the node, is the "owner" of the UUID, grant the node access to the UUID, and let the node create a volume and store the secret in the UUID). AFAICS libvirt does not currently have the concept of an "user"; as described above, such features can be handled in an external key store. If you plan to add the "user" concept to libvirt, this would definitely make sense. Mirek