Hi, QEMU offers both NBD client and server functionality. The NBD protocol runs unencrypted, which is a problem when the client and server communicate over an untrusted network. The particular use case that prompted this mail is storage migration in OpenStack. The goal is to encrypt the NBD connection between source and destination hosts during storage migration.

I think we can integrate TLS into the NBD protocol as an optional flag. A quick web search does not reveal existing open source SSL/TLS NBD implementations. I do see a VMware NBDSSL protocol but there is no specification so I guess it is proprietary. The NBD protocol starts with a negotiation phase. This would be the appropriate place to indicate that TLS will be used. After client and server complete TLS setup the connection can continue as normal.

Besides QEMU, the userspace NBD tools (http://nbd.sf.net/) can also be extended to support TLS. In this case the kernel needs a localhost socket and userspace handles TLS. Thoughts? Stefan

On Thu, Sep 04, 2014 at 04:19:17PM +0200, Benoît Canet wrote: > The Wednesday 03 Sep 2014 à 17:44:17 (+0100), Stefan Hajnoczi wrote : > > Hi, > > QEMU offers both NBD client and server functionality. The NBD protocol > > runs unencrypted, which is a problem when the client and server > > communicate over an untrusted network. > > > > The particular use case that prompted this mail is storage migration in > > OpenStack. The goal is to encrypt the NBD connection between source and > > destination hosts during storage migration. > > I agree this would be usefull. > > > > > I think we can integrate TLS into the NBD protocol as an optional flag. > > A quick web search does not reveal existing open source SSL/TLS NBD > > implementations. I do see a VMware NBDSSL protocol but there is no > > specification so I guess it is proprietary. > > > > The NBD protocol starts with a negotiation phase. This would be the > > appropriate place to indicate that TLS will be used. After client and > > server complete TLS setup the connection can continue as normal. > > Prenegociating TLS look like we will accidentaly introduce some security hole. > Why not just using a dedicated port and let the TLS handshake happen normaly ? The mgmt app (libvirt in this case) chooses an arbitrary port when telling QEMU to setup NBD, so we don't need to specify any alternate port. I'd expect that libvirt just tell QEMU to enable NBD at both ends, and we immediately do the TLS handshake upon opening the connection. Only once TLS is established, should the NBD protocol start running. IOW we don't need to modify the NBD protocol at all. If the mgmt app tells QEMU to enable TLS at one end and not the other, the mgmt app gets what it deserves (a failed TLS handshake). We certainly would not want QEMU to auto-negotiate and fallback to plain text in this case.

Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Thu, Sep 04, 2014 at 04:19:17PM +0200, Benoît Canet wrote: > The Wednesday 03 Sep 2014 à 17:44:17 (+0100), Stefan Hajnoczi wrote : >> Hi, >> QEMU offers both NBD client and server functionality. The NBD protocol >> runs unencrypted, which is a problem when the client and server >> communicate over an untrusted network. >> >> The particular use case that prompted this mail is storage migration in >> OpenStack. The goal is to encrypt the NBD connection between source and >> destination hosts during storage migration. > > I agree this would be usefull. > >> >> I think we can integrate TLS into the NBD protocol as an optional flag. >> A quick web search does not reveal existing open source SSL/TLS NBD >> implementations. I do see a VMware NBDSSL protocol but there is no >> specification so I guess it is proprietary. >> >> The NBD protocol starts with a negotiation phase. This would be the >> appropriate place to indicate that TLS will be used. After client and >> server complete TLS setup the connection can continue as normal. > > Prenegociating TLS look like we will accidentaly introduce some security hole. > Why not just using a dedicated port and let the TLS handshake happen normaly ? The mgmt app (libvirt in this case) chooses an arbitrary port when telling QEMU to setup NBD, so we don't need to specify any alternate port. I'd expect that libvirt just tell QEMU to enable NBD at both ends, and we immediately do the TLS handshake upon opening the connection. Only once TLS is established, should the NBD protocol start running. IOW we don't need to modify the NBD protocol at all.

If the mgmt app tells QEMU to enable TLS at one end and not the other, the mgmt app gets what it deserves (a failed TLS handshake). We certainly would not want QEMU to auto-negotiate and fallback to plain text in this case. Regards, Daniel

On Thu, Sep 04, 2014 at 04:19:17PM +0200, Benoît Canet wrote: > The Wednesday 03 Sep 2014 à 17:44:17 (+0100), Stefan Hajnoczi wrote : > > Hi, > > QEMU offers both NBD client and server functionality. The NBD protocol > > runs unencrypted, which is a problem when the client and server > > communicate over an untrusted network. > > > > The particular use case that prompted this mail is storage migration in > > OpenStack. The goal is to encrypt the NBD connection between source and > > destination hosts during storage migration. > > I agree this would be usefull. > > > > > I think we can integrate TLS into the NBD protocol as an optional flag. > > A quick web search does not reveal existing open source SSL/TLS NBD > > implementations. I do see a VMware NBDSSL protocol but there is no > > specification so I guess it is proprietary. > > > > The NBD protocol starts with a negotiation phase. This would be the > > appropriate place to indicate that TLS will be used. After client and > > server complete TLS setup the connection can continue as normal. > > Prenegociating TLS look like we will accidentaly introduce some security hole.

Can you elaborate on that? How would it be a security hole? > Why not just using a dedicated port and let the TLS handshake happen normaly ? Because STARTTLS(-like) protocols are much cleaner; no need to open two firewall ports. Also, when I made the request for a port number at IANA, I was told I wouldn't get another port for a "secure" variant -- which makes sense. As such, if the reference implementation is ever going to support TLS, it has to be in a way where it is negotiated at setup time. SMTP can do this safely. So can LDAP. I'm sure we can come up with a safe way of negotiating TLS. If you want to disallow nonencrypted communication, I'm sure it can be made possible to require TLS for (some of) your exports. (my objections on userspace/kernelspace issues still stand, however) -- It is easy to love a country that is famous for chocolate and beer -- Barack Obama, speaking in Brussels, Belgium, 2014-03-26

The Friday 05 Sep 2014 à 00:07:04 (+0200), Wouter Verhelst wrote : > On Thu, Sep 04, 2014 at 04:19:17PM +0200, Benoît Canet wrote: > > The Wednesday 03 Sep 2014 à 17:44:17 (+0100), Stefan Hajnoczi wrote : > > > Hi, > > > QEMU offers both NBD client and server functionality. The NBD protocol > > > runs unencrypted, which is a problem when the client and server > > > communicate over an untrusted network. > > > > > > The particular use case that prompted this mail is storage migration in > > > OpenStack. The goal is to encrypt the NBD connection between source and > > > destination hosts during storage migration. > > > > I agree this would be usefull. > > > > > > > > I think we can integrate TLS into the NBD protocol as an optional flag. > > > A quick web search does not reveal existing open source SSL/TLS NBD > > > implementations. I do see a VMware NBDSSL protocol but there is no > > > specification so I guess it is proprietary. > > > > > > The NBD protocol starts with a negotiation phase. This would be the > > > appropriate place to indicate that TLS will be used. After client and > > > server complete TLS setup the connection can continue as normal. > > > > Prenegociating TLS look like we will accidentaly introduce some security hole. I was thinking of the fallback to cleartext case.

[Cc: to nbd-general list added] On Wed, Sep 03, 2014 at 05:44:17PM +0100, Stefan Hajnoczi wrote: > Hi, > QEMU offers both NBD client and server functionality. The NBD protocol > runs unencrypted, which is a problem when the client and server > communicate over an untrusted network. > > The particular use case that prompted this mail is storage migration in > OpenStack. The goal is to encrypt the NBD connection between source and > destination hosts during storage migration. I've never given encrypted NBD high priority, since I don't think encryption without authentication serves much purpose -- and I haven't gotten around to adding authentication yet (for which I have plans; but other things have priority).

> I think we can integrate TLS into the NBD protocol as an optional flag. > A quick web search does not reveal existing open source SSL/TLS NBD > implementations. I do see a VMware NBDSSL protocol but there is no > specification so I guess it is proprietary. > > The NBD protocol starts with a negotiation phase. This would be the > appropriate place to indicate that TLS will be used. After client and > server complete TLS setup the connection can continue as normal. > > Besides QEMU, the userspace NBD tools (http://nbd.sf.net/) can also be > extended to support TLS. In this case the kernel needs a localhost > socket and userspace handles TLS. That introduces a possibility for a deadlock, since now your network socket isn't on the PF_MEMALLOC-protected socket anymore, which will cause the kernel to throw away packets which are needed for your nbd connection, in hopes of clearing some memory. I suppose you could theoretically do the encryption in kernel space. Not convinced that trying TLS in kernel space is a good idea, though. I have heard of people using stunnel or the likes to pipe the NBD protocol over a secure channel, with various levels of success.

[Cc: to nbd-general list added] On Wed, Sep 03, 2014 at 05:44:17PM +0100, Stefan Hajnoczi wrote: > Besides QEMU, the userspace NBD tools (http://nbd.sf.net/) can also be > extended to support TLS. In this case the kernel needs a localhost > socket and userspace handles TLS. That introduces a possibility for a deadlock, since now your network socket isn't on the PF_MEMALLOC-protected socket anymore, which will cause the kernel to throw away packets which are needed for your nbd connection, in hopes of clearing some memory.

On 03.09.2014 18:44, Stefan Hajnoczi wrote: >Hi, >QEMU offers both NBD client and server functionality. The NBD protocol >runs unencrypted, which is a problem when the client and server >communicate over an untrusted network. This is not problem for NBD only, but for the rest of data that qemu sends over network, i.e. migration stream, VNC/SPICE, ...

>The particular use case that prompted this mail is storage migration in >OpenStack. The goal is to encrypt the NBD connection between source and >destination hosts during storage migration. > >I think we can integrate TLS into the NBD protocol as an optional flag. >A quick web search does not reveal existing open source SSL/TLS NBD >implementations. I do see a VMware NBDSSL protocol but there is no >specification so I guess it is proprietary. In case of libvirt, we have so called tunnelled migration (both spelled & misspelled :P) in which libvirt opens a local ports on both src & dst side and then sets up secured forwarding pipe to the other side. Or a insecured one if user wishes so. The only problem is that when I adapted libvirt for NBD, I intentionally forbade NBD in tunnelled migration as it requires one more data stream for which libvirt migration protocol is not ready yet. Having saidy that, I feel that libvirt is the show stopper here, not QEMU.

On Wed, Sep 03, 2014 at 05:44:17PM +0100, Stefan Hajnoczi wrote: Also, so mean of verification is required (otherwise, back to point 0 being vulnerable to sslstrip style attacks) either that the server's cert is signed with a certain (self-generated) CA certificate or that it matches a certain fingerprint. Doing it similarly on the server-side would allow hitting a 2nd bird (authentication.)

Tunneling the entire protocol inside an SSL connection doesn't fix that; if an attacker is able to hijack your TCP connections and change flags, then this attacker is also able to hijack your TCP connection and redirect it to a decrypting/encrypting proxy. I agree that preventing a possible SSL downgrade attack (and other forms of MITM) should be high on the priority list, but "tunnel the whole thing in SSL" doesn't do that.

Il 01/10/2014 22:23, Wouter Verhelst ha scritto: > Hi, > > On Fri, Sep 05, 2014 at 03:26:09PM +0200, Wouter Verhelst wrote: >> Tunneling the entire protocol inside an SSL connection doesn't fix that; >> if an attacker is able to hijack your TCP connections and change flags, >> then this attacker is also able to hijack your TCP connection and >> redirect it to a decrypting/encrypting proxy. >> >> I agree that preventing a possible SSL downgrade attack (and other forms >> of MITM) should be high on the priority list, but "tunnel the whole >> thing in SSL" doesn't do that. > > So, having given this some thought, I wanted to come up with a spec just > so that we had something we could all agree on. As part of that, I had a > look at qemu-nbd, and noticed that it uses the "oldstyle" handshake > protocol (on port 10809 by default -- ew, please don't do that). Can you use new-style handshake with a single unnamed export? Export names are a useless complication for qemu-nbd.

On Wed, Oct 01, 2014 at 10:23:26PM +0200, Wouter Verhelst wrote: > Hi, > > On Fri, Sep 05, 2014 at 03:26:09PM +0200, Wouter Verhelst wrote: >> Tunneling the entire protocol inside an SSL connection doesn't fix that; >> if an attacker is able to hijack your TCP connections and change flags, >> then this attacker is also able to hijack your TCP connection and >> redirect it to a decrypting/encrypting proxy. >> >> I agree that preventing a possible SSL downgrade attack (and other forms >> of MITM) should be high on the priority list, but "tunnel the whole >> thing in SSL" doesn't do that. > > So, having given this some thought, I wanted to come up with a spec just > so that we had something we could all agree on. As part of that, I had a > look at qemu-nbd, and noticed that it uses the "oldstyle" handshake > protocol (on port 10809 by default -- ew, please don't do that). > > I had to change the protocol incompatibly a few years back, because the > oldstyle protocol is broken by design; in the oldstyle negotiation > protocol, the server dumps all information it has on the export to the > client, and then moves on to the data negotiation phase, without waiting > for any reply from the client. This means the oldstyle protocol can't be > used for any sort of negotiation[1]. > > As such, I strongly suggest that qemu-nbd move to the newstyle protocol. Even if we added support for the newstyle protocol I don't see us being able to drop the oldstyle protocol. NBD is used during migration of block storage, and we need to be able to migrate from old QEMU to new QEMU and vica-verca, so can't just switch protocol in a new QEMU without retaining a way to use the old protocol.

Hi all, (added rjones from nbdkit fame -- hi there)

So I think the following would make sense to allow TLS in NBD. This would extend the newstyle negotiation by adding two options (i.e., client requests), one server reply, and one server error as well as extend one existing reply, in the following manner: - The two new commands are NBD_OPT_PEEK_EXPORT and NBD_OPT_STARTTLS. The former would be used to verify if the server will do TLS for a given export: C: NBD_OPT_PEEK_EXPORT S: NBD_REP_SERVER, with an extra field after the export name containing flags that describe the export (R/O vs R/W state, whether TLS is allowed and/or required). If the server indicates that TLS is allowed, the client may now issue NBD_OPT_STARTTLS: C: NBD_OPT_STARTTLS S: NBD_REP_STARTTLS # or NBD_REP_ERR_POLICY, if unwilling C: <initiate TLS handshake> Once the TLS handshake has completed, negotiation should continue over the secure channel. The client should initiate that by sending an NBD_OPT_* message. - The server may reply to any and all negotiation request with NBD_REP_ERR_TLS_REQD if it does not want to do anything without TLS. However, if at least one export is supported without encryption, the server must not in any case use this reply. There is no command to "exit" TLS again. I don't think that makes sense, but I could be persuaded otherwise with sound technical arguments. Thoughts? (full spec (with numbers etc) exists as an (uncommitted) diff to doc/proto.txt on my laptop, ...) -- It is easy to love a country that is famous for chocolate and beer -- Barack Obama, speaking in Brussels, Belgium, 2014-03-26

On Sat, Oct 18, 2014 at 07:33:22AM +0100, Richard W.M. Jones wrote: > On Sat, Oct 18, 2014 at 12:03:23AM +0200, Wouter Verhelst wrote: > > Hi all, > > > > (added rjones from nbdkit fame -- hi there) > > [I'm happy to implement whatever you come up with, but I've added > Florian Weimer to CC who is part of Red Hat's product security group] > > > So I think the following would make sense to allow TLS in NBD. > > > > This would extend the newstyle negotiation by adding two options (i.e., > > client requests), one server reply, and one server error as well as > > extend one existing reply, in the following manner: > > > > - The two new commands are NBD_OPT_PEEK_EXPORT and NBD_OPT_STARTTLS. The > > former would be used to verify if the server will do TLS for a given > > export: > > > > C: NBD_OPT_PEEK_EXPORT > > S: NBD_REP_SERVER, with an extra field after the export name > > containing flags that describe the export (R/O vs R/W state, > > whether TLS is allowed and/or required). IMHO the server should never provide *any* information about the exported volume(s) until the TLS layer has been fully setup. ie we shouldn't only think about the actual block data transfers, we should protect the entire NBD protocol even metadata related operations.

Furthermore, STARTTLS is vulnerable to active attacks: if you can get between the peers, you can make them fall back to unencrypted silently. How do you plan to guard against that?

I cannot comment on whether the proposed STARTTLS command is at the correct stage of the NBD protocol. If there is a protocol description for NBD, I can have a look.

On Tue, Oct 21, 2014 at 12:10:39AM +0200, Wouter Verhelst wrote: > On Mon, Oct 20, 2014 at 01:56:43PM +0200, Florian Weimer wrote: > > I cannot comment on whether the proposed STARTTLS command is at the correct > > stage of the NBD protocol. If there is a protocol description for NBD, I > > can have a look. > > To make this discussion in that regard a bit easier, I've committed the > proposed spec to a separate branch: > > https://github.com/yoe/nbd/blob/tlsspec/doc/proto.txt So, if I'm understanding correctly the new style "fixed" handshake currently works like this: Server starts transmission with version info - S: "NBDMAGIC" - S: 0x49484156454F5054 - S: 0x1 And the client acknowledges with a 32 bits of and first bit set - C: 0x1 Now the client has to request one or more options, of which the NBD_OPT_EXPORT_NAME (0x1) option is mandatory, so it will be the first thing the client sends next.

So it would send - C: 0x49484156454F5054 - C: 0x1 - C: 0xa (length of name) - C: "volumename" You're proposing to add a new option NBD_OPT_STARTTLS (0x5). For this to work we would need to update the language to say that the NBD_OPT_STARTTLS must be the first option that the client sends to the server, because we want the option name to transmit in ciphertext.

So the new protocol startup would be Server starts transmission with version info - S: "NBDMAGIC" - S: 0x49484156454F5054 - S: 0x1 And the client acknowledges with a 32 bits of zero - C: 0x1 Client requests TLS - C: 0x49484156454F5054 - C: 0x5 Server acknowledges TLS: - S: 0x3e889045565a9 - S: 0x5 - S: 0x1 (REP_ACK) - S: 0x0 Now the TLS handshake is performed by client + server The client can now set other options using the secure channel, of which the NBD_OPT_EXPORT_NAME (0x1) option is mandatory, so it will be the first thing the client

sends next. So they would send - C: "0x49484156454F5054" - C: 0x1 - C: 0xa (length of name) - C: "volumename" ....etc... This is secure when both client and server want to use TLS. This is secure when the client wants TLS and the server does not want TLS, because the server will reject TLS and the client will then close the connection. My concern is the case where the client does not want TLS but the server does want TLS. In this case the client will immediately send the NBD_OPT_EXPORT_NAME in clear text over the wire, not giving the server any chance to reject the use of a clear text channel.

This problem is inherant to the approach of using the NBD protocol options to negotiate TLS.

One way I see to solve that insecurity, would be for the server to make use of one of the extra reserved bits in the very first message it sends. ie we need to negotiate TLS immediately after the version number / magic acknowledgment, before we actually start the main body of the protocol ie, with the new style fixed handshake the server should send Server starts transmission with version info - S: "NBDMAGIC" - S: 0x49484156454F5054 - S: 0x2

And the client acknowledges with a 32bits and first two bits set - C: 0x2 The questionmark here is what happens when the client sees the second bit of reserved field set ? The protocol docs merely say "S: 16 bits of zero (bits 1-15 reserved for future use; bit 0 in use to signal fixed newstyle (see below))" But don't mention what clients should do upon seeing unknown bits set in the server's message ?

If clients ignore unknown bits, then we have the same problem where a client can ignore the TLS bit and start sending option name in the clear before the server rejects the session. If clients abort (close connection) on seeing unknown bits, then we are good. A 3rd alternative is to actually use a separate magic number, which should guarantee the client would immediately close upon seeing a magic number which indicated TLS is required.

Stefan Hajnoczi <stefanha(a)redhat.com&gt; writes: > On Mon, Oct 20, 2014 at 08:58:14AM +0100, Daniel P. Berrange wrote: >> On Sat, Oct 18, 2014 at 07:33:22AM +0100, Richard W.M. Jones wrote: >> > On Sat, Oct 18, 2014 at 12:03:23AM +0200, Wouter Verhelst wrote: >> > > Hi all, >> > > >> > > (added rjones from nbdkit fame -- hi there) >> > >> > [I'm happy to implement whatever you come up with, but I've added >> > Florian Weimer to CC who is part of Red Hat's product security group] >> > >> > > So I think the following would make sense to allow TLS in NBD. >> > > >> > > This would extend the newstyle negotiation by adding two options (i.e., >> > > client requests), one server reply, and one server error as well as >> > > extend one existing reply, in the following manner: >> > > >> > > - The two new commands are NBD_OPT_PEEK_EXPORT and NBD_OPT_STARTTLS. The >> > > former would be used to verify if the server will do TLS for a given >> > > export: >> > > >> > > C: NBD_OPT_PEEK_EXPORT >> > > S: NBD_REP_SERVER, with an extra field after the export name >> > > containing flags that describe the export (R/O vs R/W state, >> > > whether TLS is allowed and/or required). >> >> IMHO the server should never provide *any* information about the exported >> volume(s) until the TLS layer has been fully setup. ie we shouldn't only >> think about the actual block data transfers, we should protect the entire >> NBD protocol even metadata related operations. > > This makes sense. Seconded.

> TLS is about the transport, not about a particular NBD export. The only > thing that should be communicated is STARTTLS. Furthermore, STARTTLS is vulnerable to active attacks: if you can get between the peers, you can make them fall back to unencrypted silently. How do you plan to guard against that?

See also https://www.agwa.name/blog/post/starttls_considered_harmful

Wouter Verhelst <w(a)uter.be&gt; writes: > On Mon, Oct 20, 2014 at 01:51:43PM +0200, Markus Armbruster wrote:

>> Furthermore, STARTTLS is vulnerable to active attacks: if you can get >> between the peers, you can make them fall back to unencrypted silently. >> How do you plan to guard against that? > > As I've said before in this discussion, encryption downgrade attacks are > not specific to STARTTLS; as soon as you have have an "encrypted" and an > "unencrypted" variant of a protocol, that becomes a problem. After all, > if an attacker can modify the communication so that STARTTLS is filtered > out of the communication, they can most likely also redirect all traffic > to a decrypting/encrypting proxy. > > The only way to fix that is through userspace; make "opportunistic" TLS > (i.e., use it if available, but move on if not) difficult to achieve. > >> See also https://www.agwa.name/blog/post/starttls_considered_harmful > > A random blog post by an author who is speaking about STARTTLS in > general terms is not a good technical argument for why STARTTLS is a bad > idea in *this* specific case. Misunderstanding. I didn't mean to claim "STARTTLS is bad". If I wanted to say that, I would've said it directly. I was merely asking how you plan to guard against downgrade attacks. I gather your advice is to make the client (QEMU) insist on TLS, and check the server's certificate. Correct?

> If I was defining a new protocol from scratch, I might dump the whole > thing in TLS to begin with. But that's just not the case, so I have to > deal with what exists already. Yes. You can still start over on a separate port. However:

> In addition, with the current state of affairs, it is *not possible* to > swap to an NBD device if you need to pipe its data through a separate > socket than the one you're handing to the kernel. The result of that is > that you can't do TLS on a device you want to swap to. This means we > need to continue to support a protocol that can do TLS for some exports, > and plain (unencrypted) traffic for other exports, *in the same running > nbd-server instance*. I don't understand enough about NBD to challenge this argument. The sad consequence is more complexity and a larger attack surface. Out of curiosity: is this "merely" an implementation restriction, or is it more fundamental?

> I did add the NBD_REP_ERR_TLS_REQD message for a server which does not > export anything unencrypted, so that it can (after the initial few > exchanges) reply to anything except for STARTTLS with "lalala, I'm not > listening until you encrypt things". However, unless it's fine to ditch > support for swapping to an NBD device (not an option from where I'm > standing), dropping support for unencrypted communication is not an > option. That's a good idea. It doesn't make things less complex, though.

Hi Markus, On Tue, Oct 21, 2014 at 10:17:17AM +0200, Markus Armbruster wrote: > Wouter Verhelst <w(a)uter.be&gt; writes: > > On Mon, Oct 20, 2014 at 01:51:43PM +0200, Markus Armbruster wrote: [...] > >> Furthermore, STARTTLS is vulnerable to active attacks: if you can get > >> between the peers, you can make them fall back to unencrypted silently. > >> How do you plan to guard against that? > > > > As I've said before in this discussion, encryption downgrade attacks are > > not specific to STARTTLS; as soon as you have have an "encrypted" and an > > "unencrypted" variant of a protocol, that becomes a problem. After all, > > if an attacker can modify the communication so that STARTTLS is filtered > > out of the communication, they can most likely also redirect all traffic > > to a decrypting/encrypting proxy. > > > > The only way to fix that is through userspace; make "opportunistic" TLS > > (i.e., use it if available, but move on if not) difficult to achieve. > > > >> See also https://www.agwa.name/blog/post/starttls_considered_harmful > > > > A random blog post by an author who is speaking about STARTTLS in > > general terms is not a good technical argument for why STARTTLS is a bad > > idea in *this* specific case. > > Misunderstanding. I didn't mean to claim "STARTTLS is bad". If I > wanted to say that, I would've said it directly. I was merely asking > how you plan to guard against downgrade attacks. I gather your advice > is to make the client (QEMU) insist on TLS, and check the server's > certificate. Correct? My advice is to give both client and server the ability to have TLS switched on or off, and possibly (but not necessarily so, and certainly not by default) also the _ability_ to negotiate TLS if the other side supports it, while not aborting if it doesn't. > > If I was defining a new protocol from scratch, I might dump the whole > > thing in TLS to begin with. But that's just not the case, so I have to > > deal with what exists already. > > Yes. You can still start over on a separate port. However: No, I can't, at least not if I want to continue to use an assigned port without breaking backwards compatibility. When 10809 was assigned to NBD, IANA made it perfectly clear that I wouldn't get a new port number for a "secure" variant. > > In addition, with the current state of affairs, it is *not possible* to > > swap to an NBD device if you need to pipe its data through a separate > > socket than the one you're handing to the kernel. The result of that is > > that you can't do TLS on a device you want to swap to. This means we > > need to continue to support a protocol that can do TLS for some exports, > > and plain (unencrypted) traffic for other exports, *in the same running > > nbd-server instance*. > > I don't understand enough about NBD to challenge this argument. The sad > consequence is more complexity and a larger attack surface. > > Out of curiosity: is this "merely" an implementation restriction, or is > it more fundamental? Well, to some extent, every software issue is merely an implementation restriction... The problem is that in order to clear memory when your swap device is on the other side of a network connection, you need to write to said swap device, which causes TCP traffic, which means the kernel needs to read TCP ACK packets (or in the case of NBD, the reply packet to the NBD_CMD_WRITE request that you sent out) to ensure that the traffic has in fact reached its destination. Unfortunately, you can't impose ordering on incoming network traffic, so that means you may need to read through a whole lot of youtube MPEG traffic or some such in order to find your one TCP ACK that tells you your swapout has been processed correctly and you can now clear the memory which you wrote out to swap. It's fairly obvious how that could lead to a deadlock if you don't know that this one data stream is not related to swapspace while this other one here is. That's why a PF_MEMALLOC kernel-space socket option was created; sockets marked with that option are related to a swapdevice, so when the kernel is low on memory, it will throw away all incoming network traffic _except_ for packets destined for a socket marked with that option, in the hope that this will allow us to clear up some memory. Since it's a kernel-space only thing, that means you can't mark sockets as such from userspace; so if the NBD traffic is wrapped in some other traffic from which it needs to be unwrapped in userspace, then the userspace component would basically be a proxy with two sockets -- one towards the server, one towards the kernel. The socket towards the kernel would have the PF_MEMALLOC socket option set, but the one towards the server would not, and *that* is the one which actually has data flowing in over the network. There's your deadlock again. Since, with the current state of affairs, the NBD handshake is done in user space with the actual data pushing stuff later being done in kernel space, that means you do actually need to unwrap the TLS traffic in userspace. I can't think of many ways that would allow us to get around that issue: - Figure out a way to move negotiated keys from user space to kernel space, and handle the TLS in kernel space rather than user space. I'm not even sure if you _can_ do TLS in kernel space (let alone whether it's a good idea). - Move the handshake to kernel space from user space. I'm not a fan of that idea. It would also still require much of the TLS in kernel space. - Patch the kernel so that userspace can mark a particular socket with PF_MEMALLOC. And mlock() the whole TLS stack into memory (including keys etc). All three of those require "patch the kernel", so at any rate going there would require "wait until the kernel supports it before we can do TLS", which I think is not a good option. > > I did add the NBD_REP_ERR_TLS_REQD message for a server which does not > > export anything unencrypted, so that it can (after the initial few > > exchanges) reply to anything except for STARTTLS with "lalala, I'm not > > listening until you encrypt things". However, unless it's fine to ditch > > support for swapping to an NBD device (not an option from where I'm > > standing), dropping support for unencrypted communication is not an > > option. > > That's a good idea. It doesn't make things less complex, though. True. -- It is easy to love a country that is famous for chocolate and beer -- Barack Obama, speaking in Brussels, Belgium, 2014-03-26 ------------------------------------------------------------------------------ Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://p.sf.net/sfu/Zoho _______________________________________________ Nbd-general mailing list Nbd-general(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nbd-general

On Sat, Oct 25, 2014 at 12:43:35PM +0200, Wouter Verhelst wrote: > I haven't seen a reply to this anymore. Do people still have comments? > I'm planning on doing a release of nbd later this weekend, and would > like to include this (not the TLS implementation yet, but at least the > spec) Hi Wouter, From https://github.com/yoe/nbd/blob/tlsspec/doc/proto.txt: * NBD_OPT_STARTTLS (5) The client wishes to initiate TLS. XXX Data. Is there text missing for "XXX Data"?

Also, I suggest at least developing a prototype before releasing the specification changes. Issues that were unknown ahead of time might be discovered during development.

Why the rush to release specification changes?