<snip/> No, this has nothing to do with RPC. Let me "pseudocode" this: int virConnectGetAllDomainStats(virConnectPtr conn, ..., unsigned int flags) { bool legacy = flags & VIR_CONNECT_GET_ALL_DOMAIN_STATS_OK_IF_NOT_ATOMIC;

On Tue, May 23, 2017 at 17:19:53 +0200, Martin Kletzander wrote: > On Tue, May 23, 2017 at 05:07:40PM +0200, Michal Privoznik wrote: > > On 05/23/2017 04:35 PM, Martin Kletzander wrote: > > > On Tue, May 23, 2017 at 04:23:30PM +0200, Peter Krempa wrote: [...] > > > > + * Note that this API is prone to exceeding maximum RPC if querying > > > > too many VMs > > > > + * with lots of statistics. It's suggested to query in batches of > > > > 10VMs, which > > > > + * should be good enough for VMs with 3000 disks + networks. > > > > + * > > > > > > Coming to think about it... Why don't we just batch this ourselves under > > > the hood and just return the merged result? > > > > Because: > > > > https://www.redhat.com/archives/libvir-list/2017-May/msg00088.html > > > > Not on the RPC level, the API would just be syntax sugar to > virDomainListGetStats() if a flag was passed > (e.g. VIR_DOMAIN_GET_ALL_STATS_I_DONT_REALLY_CARE_IF_THIS_IS_DONE_IN_ONE_LIBVIRT_CALL) Also compared to a full fragmentation of the returned data, this would result into a worst-case-scenario memory usage of MAX_SIZE * NVMS_QUERIED_IN_ORIGINAL_CALL, when compared to an unbounded memory use of the full fragmentation approach.

-- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

On Tue, May 23, 2017 at 05:29:52PM +0200, Peter Krempa wrote: > On Tue, May 23, 2017 at 17:19:53 +0200, Martin Kletzander wrote: >> On Tue, May 23, 2017 at 05:07:40PM +0200, Michal Privoznik wrote: >> > On 05/23/2017 04:35 PM, Martin Kletzander wrote: >> > > On Tue, May 23, 2017 at 04:23:30PM +0200, Peter Krempa wrote: > > [...] > >> > > > + * Note that this API is prone to exceeding maximum RPC if >> querying >> > > > too many VMs >> > > > + * with lots of statistics. It's suggested to query in batches of >> > > > 10VMs, which >> > > > + * should be good enough for VMs with 3000 disks + networks. >> > > > + * >> > > >> > > Coming to think about it... Why don't we just batch this >> ourselves under >> > > the hood and just return the merged result? >> > >> > Because: >> > >> > https://www.redhat.com/archives/libvir-list/2017-May/msg00088.html >> > >> >> Not on the RPC level, the API would just be syntax sugar to >> virDomainListGetStats() if a flag was passed >> (e.g. >> VIR_DOMAIN_GET_ALL_STATS_I_DONT_REALLY_CARE_IF_THIS_IS_DONE_IN_ONE_LIBVIRT_CALL) >> > > Also compared to a full fragmentation of the returned data, this would > result into a worst-case-scenario memory usage of MAX_SIZE * > NVMS_QUERIED_IN_ORIGINAL_CALL, when compared to an unbounded memory use > of the full fragmentation approach. If I get what you are saying, then the same would happen if the mgmt app (or client) implemented it themselves. We would basically just provide the guessing logic.

On Wed, May 24, 2017 at 12:49:58 +0200, Michal Privoznik wrote: > On 05/24/2017 12:32 PM, Martin Kletzander wrote: > > On Tue, May 23, 2017 at 05:29:52PM +0200, Peter Krempa wrote: > >> On Tue, May 23, 2017 at 17:19:53 +0200, Martin Kletzander wrote: > >>> On Tue, May 23, 2017 at 05:07:40PM +0200, Michal Privoznik wrote: [...] > >> Also compared to a full fragmentation of the returned data, this would > >> result into a worst-case-scenario memory usage of MAX_SIZE * > >> NVMS_QUERIED_IN_ORIGINAL_CALL, when compared to an unbounded memory use > >> of the full fragmentation approach. > > > > If I get what you are saying, then the same would happen if the mgmt app > > (or client) implemented it themselves. We would basically just provide > > the guessing logic. > > That's quite exact. I mean the word 'guessing'. We can't really provide > reliable way of dealing with what you're suggesting (unless we cut the > limit really small) nor we can guarantee atomicity. Therefore I think it > would be a waste of time to work on this. Yes, it can be done, but the > benefits are pretty small IMO. Yes, I agree. Therefore I opted to add the notice for this API.

-- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

On Wed, May 24, 2017 at 12:49:58PM +0200, Michal Privoznik wrote: > That's quite exact. I mean the word 'guessing'. We can't really provide > reliable way of dealing with what you're suggesting (unless we cut the > limit really small) nor we can guarantee atomicity. Therefore I think it > would be a waste of time to work on this. Yes, it can be done, but the > benefits are pretty small IMO. Why is atomicity a problem?

Just structure the libvirtd messages so that you have: COLLECT_THE_STATS - saves the stats into an internal buffer in libvirtd and returns a handle and a number of stat items RETURN_THE_STATS - returns partial subset of previously collected stats, called multiple times to transfer the data back to libvirt FREE_THE_STATS - free the internal buffer

On Wed, May 24, 2017 at 04:16:45PM +0200, Michal Privoznik wrote: > On 05/24/2017 02:47 PM, Richard W.M. Jones wrote: >> >> On Wed, May 24, 2017 at 12:49:58PM +0200, Michal Privoznik wrote: >>> That's quite exact. I mean the word 'guessing'. We can't really provide >>> reliable way of dealing with what you're suggesting (unless we cut the >>> limit really small) nor we can guarantee atomicity. Therefore I think it >>> would be a waste of time to work on this. Yes, it can be done, but the >>> benefits are pretty small IMO. >> >> Why is atomicity a problem? > > The atomicity steps in depending on what level we are talking about > serialization. If we do it the way Martin suggested (= on public API > level) then the list of domains may change between two iterations of the > suggested loop. Thus the result obtained would not reflect the reality. > However, if we are talking on RPC level, then there's no atomicity > problem as RPC is 'blind' to the data (it doesn't care what data is sent). > >> Just structure the libvirtd >> messages so that you have: >> >> COLLECT_THE_STATS >> - saves the stats into an internal buffer in libvirtd >> and returns a handle and a number of stat items >> RETURN_THE_STATS >> - returns partial subset of previously collected stats, >> called multiple times to transfer the data back to libvirt >> FREE_THE_STATS >> - free the internal buffer > > This is exactly what I was proposing in my e-mail that I linked in the > other thread. You just wrote it more cute. Yet again, what's the > gain/advantage of this over one big message? In the libguestfs case there is an actual security concern. The daemon runs in the untrusted context of the guest, where a malicious guest filesystem or program could (in some cases quite easily) send back arbitrarily large messages to the library, tieing up infinite resources on the host. In libvirt the danger is possibly more on the other side (modified library sends infinitely large message to libvirtd). There's also an issue of libvirt connecting to a compromised remote host. Whether having a maximum message size prevents all such attacks I'm less clear about, but it probably helps against simple ones.

On Wed, May 24, 2017 at 05:12:06PM +0200, Michal Privoznik wrote: > On 05/24/2017 04:58 PM, Richard W.M. Jones wrote: >> On Wed, May 24, 2017 at 04:16:45PM +0200, Michal Privoznik wrote: >>> On 05/24/2017 02:47 PM, Richard W.M. Jones wrote: >>>> >>>> On Wed, May 24, 2017 at 12:49:58PM +0200, Michal Privoznik wrote: >>>>> That's quite exact. I mean the word 'guessing'. We can't really >>>>> provide >>>>> reliable way of dealing with what you're suggesting (unless we cut >>>>> the >>>>> limit really small) nor we can guarantee atomicity. Therefore I >>>>> think it >>>>> would be a waste of time to work on this. Yes, it can be done, but >>>>> the >>>>> benefits are pretty small IMO. >>>> >>>> Why is atomicity a problem? >>> >>> The atomicity steps in depending on what level we are talking about >>> serialization. If we do it the way Martin suggested (= on public API >>> level) then the list of domains may change between two iterations of >>> the >>> suggested loop. Thus the result obtained would not reflect the reality. >>> However, if we are talking on RPC level, then there's no atomicity >>> problem as RPC is 'blind' to the data (it doesn't care what data is >>> sent). >>> >>>> Just structure the libvirtd >>>> messages so that you have: >>>> >>>> COLLECT_THE_STATS >>>> - saves the stats into an internal buffer in libvirtd >>>> and returns a handle and a number of stat items >>>> RETURN_THE_STATS >>>> - returns partial subset of previously collected stats, >>>> called multiple times to transfer the data back to libvirt >>>> FREE_THE_STATS >>>> - free the internal buffer >>> >>> This is exactly what I was proposing in my e-mail that I linked in the >>> other thread. You just wrote it more cute. Yet again, what's the >>> gain/advantage of this over one big message? >> >> In the libguestfs case there is an actual security concern. The >> daemon runs in the untrusted context of the guest, where a malicious >> guest filesystem or program could (in some cases quite easily) send >> back arbitrarily large messages to the library, tieing up infinite >> resources on the host. >> >> In libvirt the danger is possibly more on the other side (modified >> library sends infinitely large message to libvirtd). There's also an >> issue of libvirt connecting to a compromised remote host. >> >> Whether having a maximum message size prevents all such attacks I'm >> less clear about, but it probably helps against simple ones. > > I'm not a security expert but I view both approach the same from > security POV. I mean, I'm all up for limits. Don't get me wrong there. > It's just that we have two options: > > a) one API call = one RPC call > b) one API call split across multiple RPC messages > > So, it's obvious that in case a) we need bigger limit for the RPC > message to fit all the data. In case b) the limit for the RPC message > can be smaller, but in turn we need limit for the maximum messages per > one API call. Both of these approaches consume about the same memory on > src & dst (I think option b) does consume slightly more because of all > the extra headers sent with each message, but that's not the point). > Now, if I were an attacker I can very well send one big message as well > as N small messages to fill up the buffers. Therefore I think the attack > surface is the same for both of these approaches. > > a) is easier to implement IMO. > You are forgetting one thing. If b) is only utilized on the way from the daemon to the client, then you can keep the number of messages per RPC call unlimited without any security concern. The only thing that would have to get compromised is the daemon and if that happens, there are bigger issues than DoS'ing the client. It would actually solve all of our current issues (not talking about client starting a domain with 10,000 disks, of course).

On Tue, May 23, 2017 at 05:07:40PM +0200, Michal Privoznik wrote: > Because: > > https://www.redhat.com/archives/libvir-list/2017-May/msg00088.html I don't think this is a reason at all. Libguestfs uses an RPC system which was modelled on the libvirt one, and has exactly the same problem with message size limits, except smaller -- 4MB and we've never had to increase it. We get around this by batching operations over messages as necessary (eg [1]). This adds a little complexity in the implementation of the API, but the point is that the complexity is entirely hidden to users of the APIs. Rich. [1] https://github.com/libguestfs/libguestfs/blob/master/lib/file.c#L375-L415 In this code, ‘guestfs_impl_lstatnslist’ is the publicly visible API (it's the implementation of the public API guestfs_lstatnslist). ‘guestfs_internal_lstatnslist’ is the message which travels over the RPC. The batching of 1000 requests per message was chosen based on pathname limits on Linux so that each request will always fit into a single message.

-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org -- libvir-list mailing list libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

On 05/24/2017 02:42 PM, Richard W.M. Jones wrote: > On Tue, May 23, 2017 at 05:07:40PM +0200, Michal Privoznik wrote: >> Because: >> >> https://www.redhat.com/archives/libvir-list/2017-May/msg00088.html > > I don't think this is a reason at all. > > Libguestfs uses an RPC system which was modelled on the libvirt one, > and has exactly the same problem with message size limits, except > smaller -- 4MB and we've never had to increase it. So you're basically doing what I'm describing in point a). Transforming problem to another one. The maximum number of 4MB messages. > > We get around this by batching operations over messages as necessary > (eg [1]). This adds a little complexity in the implementation of the > API, but the point is that the complexity is entirely hidden to users > of the APIs. Exactly. A little complexity. That's in your case. In our case it would be slightly more complex IMO (although I've never tried to write the code, so I cannot say really). BUT, more importantly why even bother when we can just raise the limit of the message? The limits are there so that if one side starts sending malicious packets it won't eat all the memory on the other side. Well, what if the attacker is slightly more ingenious and sends N messages that fit size limit for one message? I don't really see a difference between raising limit for one message and splitting the data into multiple messages.

Michal