The problem(s) ============== While a hypervisor agnostic API is useful for some users, it is completely irrelevant, and potentally even painful, for other users. We made some concessions to this when we introduced hypervisor specific XML namespaces and option for hypervisor specific add-on APIs. We tell apps these are all unsupported for production usage though. IOW, have this pony, but you can never play with it. The hypervisor agnostic API approach inevitably took us in a direction where libvirt (or something below it) is in charge of managing the QEMU process lifecycle. We can't expose the concept of process management upto the client application because many hypervisors don't present virtual machine as UNIX processes, or merely have processes as a secondary concept eg with Xen a QEMU process is just subservient to the main Xen guest domain concept. Essentially libvirt expects the application to treat the hypervisor / compute host as a black box and just rely on libvirt APIs for managing the virtual machines, because that is the only way to provide a hypervisor agnostic view of a commpute host. This approach also gives parity of functionality regardless of whether the management app is on a remote machine, vs colocated locally with libvirtd. Most of the large scale management applications have ended up with a model where they have a component on each compute host talking to libvirt locally over a UNIX socket, with TCP based access only really used for live migration. Thus the management apps have rarely considered the Linux OS to truely be a black box when dealing with KVM. To some degree they all peer inside the box, and wish to take advantage of some of the concepts Linux exposes to integrate with the hypervisor. The inability to directly associate a client with the lifecycle of a single QEMU process has long been a source of frustration to libguestfs. The level of indirection forced by use of libvirtd does not map well to how libguestfs wants to use QEMU. Essentially libguestfs isn't trying to use QEMU in a system mangement scenario, but rather utilize it as an embedded technology component. As a result, libguestfs still has its own non-libvirt based way of spawning QEMU which is often used in preference to its libvirt based impl. Other apps like libvirt-sandbox have faced the same. When systemd came into existance, finally providing good mechanisms for process management in Linux machines, we found a tension between what libvirt wants todo and what systemd wants todo. The best we've managed is a compromise where libvirt spawns the guest, but then registers it with systemd. Users can't directly spawn QEMU guests with systemd and then manage them with libvirt. We've not seen people seriously try to manage QEMU guests directly with systemd, but it is fair to say that the combination of systemd and docker have taken away most potential users of libvirt's LXC driver, as apps managing containers don't want to treat the host as a blackbox, they want to have more direct control. The failure to get adoption of the LXC driver serves as a cautionary tale for what could happen to use of the libvirt QEMU driver in future. More recently the increasing interest in use of containers is raising new interesting architectures for the management of processes. In particular the Kubernetes project can be considerd to provide cluster-wide management of processes, aka k8s is systemd for data centers. Again there is interest in using Kubernetes to manage QEMU guests across the data center. The KubeVirt project is attempting to bridge the conflicting world views of libvirt and Kubernetes to build a KVM management system to eventually replace both oVirt and OpenStack. The need to have libvirtd spawn the QEMU processes is causing severe complications for KubeVirt architecture, causing them to seriously consider not using libvirt for management KVM. This issue is a major blocking item for KubeVirt to the extent that they may well have to abandon use of libvirt to get the process startup & resource integration model they need. On Linux as far as hypervisor technology is concerned, KVM has won the battles and the war. OpenStack user surveys have constantly put KVM/QEMU on top with at least one order of magnitude higher usage than any other technology. Amazon was always the major reference for usage of Xen in public cloud and even they appear to be about to pivot to KVM. IOW, while providing a hypervisor agnostic management API is still a core competancy of libvirt, we need to embrace the reality that KVM is the defacto standard on Linux and better enable people to take avantage of its unique features, because that is where most of our userbase is. A second example of limitations of the purely hypervisor agnostic approach to libvirt is the way our API design is fully synchronous. An application calling a libvirt API blocks until its execution is complete. This approach was originally driven by the need to integrate directly with various Xen backend APIs which were also mostly synchronous in design. Later we added other hypervisor targets which also exposed synchronous APIs. In parallel though, we added the libvirtd daemon for running stateful hypervisor drivers like QEMU, LXC, UML, and now Xen. We speak to this over an RPC system that can handle arbitrarily overlapping asynchronous requests, but then force it into our synchronous public API. For applications which only care about using KVM, the ability to use an asynchronous API could be very interesting as it would no longer force them to spawn large numbers of threads to get parallel API execution. The Solution(s) =============== Currently our long term public stability promise just covers the XML format and library API. To enable more interesting usage of hypervisor specific concepts it is important to consider how to provide other options beyond just the current API and XML formats. IOW, I'm not talking about making QMP command passthrough or CLI arg passthrough fully supported features, as libvirt's API & XML abstraction has clear value there. Rather I'm thinking about more architectural level changes. In particular I want to try to break down the black box model of the host, to make it possible to exploit KVM's key distinguishing feature, which is that the guest is just a normal process. An application that knows how to spawn & reap processes should be able to launch KVM as if it was just another normal process. This implies that the application needs the option to handle the fork+exec of KVM, instead of libvirt, if it so wishes. I would anticipate a standalone process "libvirt-qemu" that an application can spawn, providing a normal domain XML file via the command line or stdin. It would then connect to libvirtd to register its existance and claim its ownership of the guest name + UUID. Assuming that succeeds, 'libvirt-qemu' would directly spawn QEMU. In this manner, the QEMU process automatically inherits all the characteristics of the application that invoked the 'libvirt-qemu' binary. This means it shares the user / group ID, the security context, the cgroup placement, the set of kernel namespaces, etc. Libvirt would honour these characteristics by default, but also have ability to further refine them. For example, it would honour the initial process CPU pinning, but could still further pin individual QEMU threads. In the initial implementation I would anticipate that libvirtd still retains control over pretty much every other aspect of ongoing QEMU management. ie libvirtd still owns the monitor connection. This means there would be some assumptions / limitations in functionality in the short term. eg it might be assumed that while libvirtd & libvirt-qemu can be in different mount namespaces, they must none the less be able to see the same underlying storage in their respective namespaces. The next mail in this series, however, takes things further to move actual driver functionality into libvirt-qemu, at which point limitations around namespaces would be largely eliminated. This design would solve the single biggest problem with managing QEMU from apps like libguestfs, systemd and KubeVirt. To avoid having 2 divergant launch processes, when libvirtd itself launches a QEMU process, it would have to use the same "libvirt-qemu' shim to do so. This would ensure functional equivalance regardless of whether the management app used the hypervisor agnostic API, or instead used the QEMU specific approach of running "libvirt-qemu". We made a crude attempt previously to allow apps to run their own QEMU and have it managed by libvirt, via the virDomainQemuAttach API. That API design was impossible to ever consider fully supported, because the mgmt application was still in charge of designing QEMU command line arguments, and it is impractical for libvirt to cope with an arbitrary set of args. With the new proposal, we're still using the current libvirt code for converting XML into QEMU args, so have a predictable configuration for QEMU. Thus the new approach can provide a fully supported way for applications to spawn QEMU. This concept of a "libvirt-qemu" shim is not all that far away from the current "libvirt-lxc" shim we have. With this in mind, it would also be desirable to make that a fully supported way to spawn LXC processes, which can then be managed by libvirt. This would make the libvirt LXC driver more interesting for people who wish to run containers (though it is admittedly to late to really recapture any significant usage from other container technologies). As mentioned earlier, if an application is only concerned with managing of KVM (or other stateful drivers running inside libvirtd), we have scope to be able to expose a fully asynchronous management API to applications. Such an undertaking would effectively mean creating an entirely new libvirt client library, to expose the asynchronous design, and we obvious have to keep the current library around long term regardless. Creating a new library would also involve creating new language bindings, which just adds to the work. Rather than undertake this massive amount of extra work, I think it is worth considering declaring the RPC protocol to be a fully supported interface for applications to consume. There are already projects which are re-implemented the libvirt client API directly ontop of the RPC protocol, bypassing libvirt.so. We have always strongly discouraged this, but none the less it has happened. As we have to maintain strong protocol compatibility on the RPC layer, it is effectively a stable API already. We cannot ever change it in incompatible manner without breaking our own client library implementation. So declaring it a formally supported interface for libvirt would not really involve any significant extra work on our part, just acknowledgement of the existing reality. It would perhaps involve some documentation work to assist developers wishing to consume it though. We would also have to outline the caveats of taking such an approach, which principally involve loosing ability to use the stateless hypervisor drivers which all live in the libvirt library. This is not a real issue though, because the people building ontop of the RPC protocol only care about KVM. Another example where exposing a KVM specific model might help is wrt live migration, specifically the initial launch of QEMU on the target host. Our libvirt migration API doesn't given the application direct control over this part which has caused apps like OpenStack to jump through considerable hoops when doing live migration. So just as an application should be able to launch the initial QEMU process, it should be able to directly launch it ready for incoming migration, and then trigger live migration to use this pre-launched virtual machine. In general the concept is that although the primary libvirt.so API will still consider the virt host to be a black box, below this libvirt should not be afraid to open up the black box to applications to expose hypervisor specific details as fully supported concepts. Applications can opt-in to using this, or continue to solely use the hypervisor agnostic API, as best fits their needs. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Wed, Nov 15, 2017 at 05:57:45PM +0000, Richard W.M. Jones wrote: Yes, that is pretty much exactly what I am suggesting. The libvirt-qemu would register itself with libvirtd, and then libguestfs would have to speak to libvirtd for ongoing management. Though for the purposes of shutdown, it would be valid to just kill() the children directly if desired. In the second mail in this series, I describe a way to decompose libvirtd, whereupon ongoing management could be handled inside libvirt-qemu itself. That would potentially avoid the need for libguestfs to talk to libvirtd at all. Though this would be a secondary piece of work Initially we could avoid the raciness if libvirt-qemu implemented the systemd startup notification protocol. That would let libvirt-qemu notify libguestfs (or whoever spawns it), /after/ it has successfully registered itself with libvirtd. So you can then virDomainLookupByUUID without any race (ie you can then assume that if virDomainLookupByUUID fails, it means the new QEMU has already quit) Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Sun, Nov 19, 2017 at 10:21:32PM +0100, Martin Kletzander wrote: I guess there's a distinction between what the apps has to do vs what the shim has todo. At least in the short-medium term, the shim itself would still need to communicate with libvirtd (or other related daemons if it needs to resolve other objects. eg resolve the "default" virtual network). The application does not neccessarily need to talk to libvirtd though. It would be enough to spawn the libvirt-qemu shim, and query stats out of cgroup directly and then kill() the shim when done. If you want to dynamically make changes to the QEMU config on the fly though, then we need some kind of API, and I'm not seeing a compelling reason to change the API we currently provide apps. It could, however, be possible for that libvirt.so remote driver to connect directly to the shim todo its work. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Thu, Nov 23, 2017 at 11:32:13AM +0100, Michal Privoznik wrote: That's not really a problem - I certainly expected that to be the case. There would be a '--connect URI' arg, but since this is a QEMU specific process, it could sensibly default to qemu:///session or qemu://system as appropriate Yes. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Thu, Nov 23, 2017 at 11:32:19AM +0100, Michal Privoznik wrote: NB, docs declaring this supported were already acked & merged: https://libvirt.org/support.html#rpcproto Nothing in the server can rely on the client library performing checkins before the RPC is sent, so those client side checks are at best an optimization to get quicker / clearer error message. An alternative impl would of course need to implement SASL / TLS in the compatible manner. The remoteDomainCreate() impl was a hack, because we forgot that we needed to update the virDomainPtr with the ID value. An implementing the client directly will not have any virDomainPtr object to update. They'll have their own concept there, which may not even care about having an ID value cached locally - I certainly wouldn't bother if we wrote libvirt again - using the UUID exclusively is a much better choice. So they need not have multiple RPC calls in the same place. On the other hand, since the app is not constrained by the need to follow the libvirt public client API, then may well write their client impl in a manner that doesn't have a 1-1 mapping to RPC messages - the 1-1 mapping is merely how we chose todo it for libvirt.so. They could write something higher level that triggers many RPC calls for one application call if they so desire. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Thu, Nov 23, 2017 at 04:58:55PM +0100, Michal Privoznik wrote: Agreed, I would only want our primary RPC protocol considered supported, Never any of the inter-daemon communiction protocols, which should remain subject to change - this is another good reason to only ever support UNIX domain sockets in these modular daemons - explicitly means we don't have to care about compat across libvirt versions. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On 11/14/2017 06:26 PM, Daniel P. Berrange wrote: So what about remote connections? Say hostA is running my KVMs and hostB is where mgmt app lives. If hostB is connecting to hostA's libvirt I guess it's still going to be libvirtd which will then multiplex RPC calls and redirect them to correct daemon? IOW, if hostB calls virStoragePoolGetXMLDesc() how is this request going to end up at hostA's virstoraged? Michal

The Problem(s) ============== When libvirt was created, C was the only viable choice for anything aiming to be a core system library component. At that time 2005, aside from C there were common choices of Java, Python, Perl. Java was way too heavy for a low level system component, Python was becoming popular but not widely used for low level system services and Perl was on a downward trend. None of them are accessible to arbitrary languages as libraries, without providing a RPC based API service. As it turns out libvirt did end up having RPC based approach for many virt drivers, but the original approach was to be a pure library component. IOW it is understandable why C was chosen back in 2005, but 12 years on the world around us has changed significantly. It has long been accepted that C is a very challenging language to write "safe" applications. By "safe" I mean avoiding the many problems that lead to critical security bugs. In particular the lack of a safe memory management framework leads to memory leaks, double free's, stack or heap corruption and more. The lack of strict type safety just compounds these problems. We've got many tools to help us in this area, and at times have tried to design our APIs to avoid problems, but there's no getting away from fact that even the best programmers will continually screw up memory management leading to crashes & security flaws. It is just a fact of life when using C, particularly if you want to be fast at accepting new feature proposals. It is no surprise that there have been no new mainstream programming languages in years (decades) which provide an inherantly unsafe memory management framework. Even back in 2005 security was a serious challenge, but in the last 10+ years the situation has only got worse with countless high profile security bugs a direct result of the choice to use C. Given the threat's faced today, one has to seriously consider the wisdom of writing any new system software in C. In another 10 years time, it would not surprise me if any system software still using C is considered an obsolete relic, and ripe for a rewrite in a memory safe language. There are long term implications for the potential pool of contributors in the future. There has always been a limited pool of programmers able todo a good job in C, compared to those who know higher level languages like Python/Java. A programmer write bad code in any language, but in C/C++ that bad code quickly turns into a serious problem. Libvirt has done ok despite this, but I feel our level of contribution, particularly "drive by" patch submissions, is held back by use of C. Move forward another 10 years, and while C will certainly exist, I struggle to imagine the talent pool being larger. On the contrary I would expect it to shrink, certainly in relative terms, and possibly in absolute terms, as other new languages take C's place for low level systems programming. 10 years ago, Docker would have been written in C, but they took the sensible decision to pick Go instead. This is happening everywhere I look, and if not Go, then Rust. We push up against the boundaries of what's sane todo in C in other ways too. For portability across operating systems, we have to rely on GNULIB to try to sanitize the platform inconsistencies where we use POSIX, and assume that any 3rd party libraries we use have done likewise. Even then, we've tried to avoid using the platform APIs because their designs are often too unsafe to risk using directly (strcat, malloc, free), or are not thread safe (APIs lacking _r variants). So we build our own custom C platform library on top of the base POSIX system, re-inventing the same wheel that every other project written in C invents. Every time we have to do work at the core C platform level, it is diverting time away from doing working managing higher level concepts. Our code is following an object oriented design in many areas, but such a notion is foreign to C, so we have to bolt a poor-mans OO framework on the side. This feeds back into the memory safety problem, because our OO invention cannot be type checked reliably at compile time, making it easy to do unsafe things with objects. It relies on reference counting because there's no automatic memory management. The other big trend of the past 10 years has been the increase in CPU core counts. My first libvirt dev machine had 1 physical CPU with no cores or threads or NUMA. My current libvirt dev machine has 2 CPUs, each with 6 cores, for 12 logical CPUs. Common server machines have 32/64 logical CPUs, and high end has 100's of CPUs. In 10 years, we'll see high end machines with 1000's of CPUs and entry level with mere 100's. IOW good concurrency is going to be key for any scalable application. Libvirt is actually doing reasonably well in this respect via our heavily threaded libvirtd daemon. It is not without cost though with ever more complex threading & locking models, which still have scalability problems. Part of the problem is that, despite Linux having very low overhead thread spawning, threads still consume non-trivial resources, so we try to constrain how many we use, which forces an M:N relationship between jobs we need to process and threads we have available. The Solution(s) =============== Two fairly recent languages, Go & Rust, have introduced new credible options for writing systems applications without sacrificing the performance of C, while achieving the kind of ease of use / speed of development seen with languages like Python. It goes without saying that both of them are memory safe languages, immediately solving the biggest risk of using C / C++. The particularly interesting & relevant innovation of Go is the concept of Goroutines for concurrent programming, which provide a hybrid kernel/userspace threading model. This lowers the overhead of concurrency to the point where you can consider spawning a new goroutine for each logical job. For example, instead of having a single thread or limited pool of threads servicing all QEMU monitor sockets & API clients, can you afford to have a new goroutine dedicated to each monitor socket and API client. That has the potential to dramatically simplify use of concurrency while at the same time allowing the code to make even better use of CPUs with massive core counts. It of course provides a cross platform portable core library of features, and has a massive ecosystem of developers providing further 3rd party libraries for a wide variety of features. This means developers can focus more time on solving the interesting problems in their application space. The Go code is still low level enough that it can interface with C code easily. FFI calls to C APIs can be made inline in the Go code, with no need to switch out to write a low level binding in C itself. In many ways, Go can be said to have the ease of use, fast learning & safety of Python, combined with the expressiveness of C. IOW it is a better C than C. I don't have direct experiance in Rust, but it has the same kind of benefits over C as Go does, again without the downsides of languages like Python or Java. There are some interesting unique features to Rust that can be important to some apps. In particular it does not use garbage collection, instead the user must still do manual memory management as you would with C/C++. This allows Rust to be used in performance critical cases where it is unacceptable to have a garbage collector run. Despite a requirement for manual allocation/deallocation, Rust still provides a safe memory model. This approach of avoiding abstractions which will introduce performance overhead is a theme of Rust. The cost of such an approach is that development has a higher learning curve and ongoing cost in Rust, as compared to Go. I don't believe that the unique features of Rust, over Go, are important to the needs of libvirt. eg while for QEMU it would be critical to not have a GC doing asynchronous memory deallocation, this is not at all important to libvirt. In fact precisely the opposite, libvirt would benefit much more from having GC take care of deallocation, letting developers focus attention other areas. In general, as from having a memory safe language, what libvirt would most benefit from is productivity gains & ease of contribution. This is the core competancy of Go, and why it is the right choice for usage in libvirt. The obvious question / difficulty is deciding how to adopt usage of a new language, without throwing everything away and starting from scratch. It needs to be possible for contributors to continue working on every other aspect of the project while adoption takes place over the long term. Blocking ongoing feature work for prolonged periods of time is not acceptable. There is also a question of scope of the work. A possible target would be to aim for 100% elimination of C in N years time (for a value of N that is certainly greater than 5, possibly as much as 10). There is a question of just whether that is a good use of resources, and even practical. In terms of management of KVM guests the bulk of ongoing development work, and complexity is in the libvirtd daemon. The libvirt.so library merely provides the remote driver client which is largely stable & unchanging. So with this in the mind the biggest benefits would be in tackling the daemon part of the code where all the complexity lives. As mentioned earlier, Go has a very effective FFI mechanism for calling C code from Go, and also allows Go code to be called from C. There are some caveats to be aware of with passing data between the languages, however, generally it is neccessary to copy data structures as C code is not permitted to derefence pointers that are owned by the Go GC system. There are two possible approaches to take, which can be crudely described as top down, or bottom up. In the top down approach, the C file providing the main() method gets replaced by a Go file providing an equivalent main() method, which then simply does an FFI call to the existing libvirt C APIs to run the code. For example it would just call virNetServer APIs to setup the RPC layer. Effectively have a Go program where 90% of the code is an FFI call to existing libvirt C code. Then we would gradually iterate downwards converting increasing areas of C code to Go code. In the bottom up approach, the program remains a C program, but we built .a files containing Go code for core pieces of functionality. The C code can thus call into this archive and end up executing Go code for certain pieces. Then we would gradually iterate upwards converting increasing areas of C code to Go code, until eventually reaching the top main() method. Or a hybrid of both approaches can be taken. Whichever way is chosen is going to be a long process and many bumps in the road. The best way to start, however, is probably to focus on a simple self-contained area of libvirt code. Specifically attack the virtlockd, and/or virtlogd daemons, converting them to use Go. This still need not be done in a "big bang". A first phase would be to develop the server side framework for handling our RPC protocol deserialization. This could then just dispatch RPC calls to the existing C impls. As a second phase, the RPC method impls would be converted to Go. Both of these daemons are small enough that the conversion would be possible across the time of a couple of releases. The hardest bit is likely ensuring compatibility for the re-exec() upgrade model they support, but this is none the less doable. The lessons learned in this would go a long way towards informing the best way to tackle the bigger task of the monolithic libvirtd (or equivalently the swarm of daemons the previous proposal suggests) Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Wed, Nov 15, 2017 at 12:28:30PM +0100, Bjoern Walk wrote: I don't think that is the case unless the code is essentially feature complete and not writing significant new code. That is certainly not the case with libvirt, which shows no sign of slowing down in terms of features we must develop. As long as you are continuing to write non-trivial C code, you are continuing to introduce security bugs. Further, things that in the past were not considered security flaws, and increasingly be classed as security flaws. What has improved is that we use more tools to detect the security flaws & crashing bugs after we've introduced them. eg coverity tells us about many screw ups. It is none the less an ongoing issue. I don't think that this is a sensible comparison. For libvirt we have effectively given up and said that a large portion of our APIs have semantic privileges equivalent to a root shell. This turn means that when libvirtd crashes, we claim that it is not a security flaw. We have many 100's of crashes we've solved, and their frequency of occurrance is not slowing down. IOW, if we actually classified all our crashes as CVEs we would have orders of magnitude more CVEs. Essentially we are ignoring the scale of the problem. There are countless "standard" libraries all with their own quirks. Many of them don't even really address the core safety problems. eg we could consider glib2 for libvirt, but it just replicates the same awful unsafe malloc API style that the stdc lib has, with marginal improvement. The same is true of most other standard libraries. There's essentially no chance of C ever getting a widely adopted "standard" library at this point. The only place there's any standardization work is at the POSIX level and that's waaaaay to low level. Only if we stop delivering new features. In pretty much every monthly release cycle we break platform portability at least once. We have finally have CI coverage to help us detect with this happens, but you still have todo the work to make our code portable. We've got 6000 lines of m4 code for configure.ac in libvirt to cope with this stuff. If you add in gnulib's m4 code that's 45,000 lines of m4. If there's one thing C programmers all agree on it is that autoconf is horrific to write and maintain. This in turn feeds into 1000's of #ifdef statements littered throughout the code, which leads to ongoing maintainence work. Pretty all this work simply doesn't exist in any modern language which provides an inherantly portable core runtime. Going from C to C++ is a waste of time IMHO. It just gives you a different and often more complicated ways to shoot yourself in the foot, solving very few of the core problems with C. You're still using an memory unsafe language, you still have tonnes of work to write portable code. Depending on the choice of language this is certainly a valid concern. It is the primary reason why I would not suggest use of Rust, because I think that really does limit the pool. With Go though, any C developers who has written non-trivial C programs easily has the skills to learn Go code in a matter of days. It would not be idiomatic Go code, but it doesn't take much more experiance to learn the Go mindset and start writing high quality code. Go has been tailored specifically to make the on-ramp as easy as possible. Mostly you just spend time googling the API docs to learn about the useful stadnard library APIs you can leverage. So I don't thnk you limit the contributor pool to the intersection of developers of the two languages. You really do broaden the pool to more like the union of the two sets. As mentioned, it would be absolutely critical to ensure that ongoing feature work can still take place during any transition. That is why I am not suggesting it as a "big bang" moment. It would be an incremental job over as long as 5 years or more. There will still be some periods of short term pain, but over the long term the improved productivity would easily make up for that. The recent major rewrite of firefox is a great example of this - they continued releasing on a regular schedule with significant new features, while in parallel rewriting a large part of their code in Rust. The end result is a tremendous step forward for the project.. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

As a point of reference, libguestfs started to allow OCaml in the daemon this summer, so effectively you can now use OCaml to implement libguestfs APIs. (I'm not suggesting libvirt should use OCaml). Some points: - We have a mixed C / OCaml daemon. The vast majority of the code is still C and will most likely always be in C. - This is an incremental approach. We can add new APIs in OCaml, or convert existing APIs from C to OCaml, but mostly existing APIs are left alone. - Our use of a generator makes this easier. The complexity is embedded in the generator. Developers just need to select whether an API is written in C or OCaml, and of course implement/ reimplement the API in the chosen language, and the generator handles the heavy lifting. - OCaml-implemented APIs are less code & safer, while being just as fast. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW

On Thu, Nov 16, 2017 at 02:58:33PM +0100, Peter Krempa wrote: I have a hard time calling Go ugly when the comparison is C. The things we do with C and macros are horrific by comparison. Go has very strong readability. Variable declartions syntax has little to no impact on ability to use the language productive manner. Likewise API nameing is just a matter of taste. Sure 'dial' is odd, but there's 1000's of wierdly named methods across APIs used in any language. I think evidence of libvirt and every other non-trivial C program sshows that when it comes to memory management mistakes it is essentially to eliminate the common mistakes. People make the same mistake over & over again, even experts in the language. We have tools that help us spot some of them, but nowhere near all of them, as evidenced by the constant stream of crashes we have to fix in libvirt which coverity doesn't tell us about. I agree that subtle logic problems can exist in any language and libvirt has its fair share of those and they won't be magically solved by a different language. We waste a hell of alot of resources on dealing with problems inherant in the usage of C (whether platform portability, or mmemory management or reinventing an object system). By not wasting that time, we will ultimately be able to put more time into identifying and fixing the subtle logic problems. I believe it will be a major net win. We have many 10's of thousands of lines of autoconf, and automake code, and code littered if #ifdefs many combinations of which are never tested properly. We have 65,000 lines of code just for parsing & formatting XML. The vast majority of that can be automatically handled in Go with mere annotations on struct fields, and the result is far safer because it would get XML entity escaping right without any effort. ON networking stuff we've written alot of code to help us integrate with sockets, TLS, SSH and so on, because while there are libraries for this none of the them provide the same API. Again the need to write that kind of boilerplating all goes away because there's a proper extensible I/O framework that. The amount of time we spend writing infrastructure in C that already exists in other language's standard libraries is very significant. I'm not suggesting to unconditionally reject people's contributions in C. If the area of code they want to contribute to is still using C, then they should certainly write submit patches in the same way. This is a key reasn why I say we shouldn't attempt any "stop the world & rewrite" big bang - it if far too distruptive and prevents people doing other useful work in parallel. A conversion would have to be incrementally evolved over a prolonged period of time, so we can still satisfy ongoing feature requests & bug fixes in parallel. This is not about using a cooler language, it is about using a language which will let us provide a much more reliable libvirt which doesn't randomly crash & leak memory, and where we can focus our dev resources on solving the interesting problems that libvirt has, instead of wasting time dealing with the problems of C. That is certainly a fair point. The flipside is that a great many of the reasons for needing GDB / valgrind / coverity / etc in C are removed by virtue of having a safe memory management design. In the intermediate where we have a mix of C and Go in the same process though, I accept it would make some debugging tasks harder. I don't think that's a show stopping problem, because over the long term it'd be a net win. I don't know if I'll change your mind, but I really encourage people to actually look beyond the obvious surface differences. As languages go it is explicitly designed to be easy to understand & familiar for C developers, much more so than any other competing languages out there. By having a truly compiled language, as opposed to a VM like Java, or a interpretor like python/perl/ruby/etc, you're still close to the machine layer as you are in C. You can still easily call into C APIs where you need to with ease as the basic scalar data types are all very similar to C, while giving you much easier to use compound data types (lists, strings, maps, etc). Being free from the maddness of manual memory allocation/deallocation/overruns greatly improves productivity and reliability of the code, and you no longer waste so much time reinventing the wheel whether for cross-platform porting or basic library APIs. I struggle to think of any compelling reason to continue to use C, aside from the fact that it is what I've always worked with for ~20 years. Many languages have come up in that time, but none has been able to credibly replace C for low overhead systems programming work until Go and Rust came along. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On 11/16/2017 03:55 PM, John Ferlan wrote: Goroutines are not strictly 1:1 mapped to an OS thread...it's an N:M mapping where a blocking call in a goroutine will not block any other goroutines. Modern Go defaults to a number of OS threads equal to the number of cores. Chris

"Daniel P. Berrange" <berrange(a)redhat.com&gt; writes: [...] Does this work even when the stack limit is exceeded in a C function? [...]

On 11/17/2017 06:37 AM, Daniel P. Berrange wrote: Based on https://github.com/golang/go/blob/master/src/runtime/cgo/gcc_linux_amd64.c it looks like they don't explicitly specify a stack size, at least on linux. Are there limits as to what you're allowed to do in C code called from Go? Can you fork processes, spawn threads, call setjmp/longjmp, handle signals, sleep, etc.? Chris

On Tue, Nov 14, 2017 at 05:27:01PM +0000, Daniel P. Berrange wrote: [...] Even though I agree with you on most of the opinions of this thread, I must engage in a discussion on this part. I started working on libvirt because I liked the fact that it is in C and needs to work with the low-level concepts and interfaces. I like python for rapid prototyping, but I used to be against new, "hype" languages that were popping up here and there. I must say, on the other hand, that I was surprised how choice of language can influence the contributor flow (mostly incoming). However I heard about this mostly happening in the ruby and node.js communities. Along the way, I quickly stopped thinking about Rust and Go as hype languages. Well, so much for my background... I'm going to approach the elephant in the room, but I'm not the one to defend the opinion as my understanding, as well as experience, is limited. On the other hand, that could show you that it is not that hard for a newbie to get the hang of it. I'm saying this because I might get some things wrong, but I expect the wrong information to be related mainly to Go. So the first thing I disagreed on is that in Rust you do manual allocations. In fact, you don't. Or, depending on the point of view, you do less or the same amount of manual allocation than in Go. What is the clear win for Rust it the concept of ownership and it's related to the allocation mentioned before. I am standing strongly behind the opinion that the learning curve of Rust is definitely worth it. And coming from the C world, it is easy to understand. To me, it is very easy to explain that concept to great detail to someone who has background in libvirt. And the big benefit (and still a huge opportunity for improvement WRT optimizations) is that the compiler must know about it and so it is resolved compile-time. Dereferencing or destructors are run at the end of their scope, automatically. You can nicely see that when realizing that Rust doesn't need any `defer` as Go has. Sure, Rust doesn't have green threads implementation, however there should be support for that in some library. I used vague wording due to the fact that I haven't yet found it, however it is mentioned in official docs. Apparently it was removed from the standard library due to requirements on runtime size. Rust's slogan (or one of them?) is "fearless concurrency". It builds upon the ideas from Go, however, having more information it can work with it better. It could help us considering how many problems we have with reference counting and similar. So much for defending Rust. Long story short, I think we could benefit a tiny (well, IMHO not really tiny) bit more if we go that route instead. Now for some more opinions I have. Stay with me. Not considering the Linux kernel, which has their own address sanitizer, randomizer and god-knows-whatifier, there is still plethora or projects that are successful in C and I don't see how they would need to adapt or die. I know we have bunch of stuff that needs to be fixed and the current approach doesn't make it easy. I, myself, thought about proposing new version of libvirt, which would be a rewrite of the current one. Thanks to that I have some reasons why I didn't do that. One of the things is that the kinks we have can be ironed out in C as well. It might be easier in other languages, but it is harder when you have to switch to one. We have bunch of code dealing with backwards compatibility. And I argue that this is something that causes issues on its own. What's even worse, IMHO, is that we are so much feature-driven that there is no time for any ironing. I see too much potential for refactoring in various parts of libvirt that will never see the lights of day because we need X to be implemented. And contributors sending feature requests that they fail to maintain later don't help much with that. Maybe we could fix this by saying the next Y releases will just be bugfix releases. Maybe we could help bringing new contributors by devoting some of our time to do an actual change that will make them want to help us more. I know some of you will be sick and tired hearing about Rust once more, but have you heard about how much their community is inclusion-oriented? I guess what I'm trying to say is that there are other (and maybe less disruptive) ways to handle the current problems we are facing. And then there are the "issues" with Go (and unfortunately some with Rust as well :'( ). Lot of the code for libraries is written with permissive licences, but if there is some that is LGPL-incompatible we can't use them. And in ecosystems such as Rust and Go there are fewer alternatives, so we might not find one that we'll be able to use. If that happens, there goes bunch of our time. Like nothing. How do we deal with problems/bugs in dependence libraries? I know, all the projects are pretty new, so they might be nicer to contributors. If they are not, we might need to fork or rewrite the code. Bam, another chance of losing workforce. If I may, I will ask some things about Go that I'm not that familiar with and I think they are pretty important. How does Go handle updates in dependency libs? Does it automatically pull newest version from public repositories where some unknown person can push whatever they want? Or can they be hash- or version-bound? The build process is that all the binaries are static, right? Or all the go code is static and it only has dynamic dependencies on C libraries? In such a project as libvirt, wouldn't that mean that the processes we run will be pretty heavy-weight? How is it with rebuilding after a small change. I know Go is good when it comes to compilation times. That might be something that people might like a lot. Especially those who are trying to shave off every second of compilation. However if you cannot use ccache and you always need to rebuild everything, it might increase the build-time quite a lot, even thought indirectly. Since there were some hateful opinions about Go, let me add my share as well. Just so we are on the same page and I don't miss saying anything. Not that these opinions would be as important as the ones above. You can't have /tmp mounted with noexec option unless you have TMPDIR set to some other directory. And sometimes not even in that case. I guess non-issue for some distros, but bunch of people deal with that and if seems like something that could be taken care of in Go itself and it is just not. You can't just clone a repo, cd into it and build it. You have to get the dependencies in a special manner, for that you have to have GOPATH set and based on that you have to have your directories setup similarly to what Go expects, then you need GOBIN if you want to build something and other stuff that's just not nice and doesn't make much sense. At least for newcomers. Simply the fact that it seems to me like Go is trying to go against the philosophy of "Do one thing and do it well". I know everyone is about having the build described in the same language as the project, but what comes out of it is not something I prefer. I'm not against using another language to make some stuff better. I guess it is kind of visible from the mail that I like Rust, but I'm not against other languages as well. I just want this to be full on discussion and I want my opinion to be expressed. Thanks for _listening_ ;) Have a nice day, Martin

On Mon, Nov 20, 2017 at 03:25:33PM +0000, Daniel P. Berrange wrote: This is kind of old, that code wouldn't run with newer Rust. I guess that is from far ago when it was not stabilized at all. It is a bit smarter now. The fact that you have control over when the value is getting freed is true, however you rarely have to think about that. What's more important is that the compiler prevents you from accessing value from multiple places or not knowing who "owns" (think of it as "who should take care of freeing it") the variable. If you give the ownership to someone you can't access it. The difference I see is that if you access it after some other part of the code is responsible for that variable in Rust the compiler will cut you off unless you clearly specify how the memory space related to the variable is supposed to be handled. In Go it will just work (with potential bug) but it will not crash because GC will not clean it up when someone can still access it. Granted this is usually problem with concurrent threads/coroutines (which I don't know how they handle access concurrent access in Go). Also, for example Rust doesn't allow you to have value accessible from multiple threads unless it is guarded by thread-safe reference counter and does not allow you to modify it unless it is guarded by Mutex, RWLock or s one of the Atomic types. Again, I don't know Go that much, I'm still yet to delve into the deep unknowns of it, but I haven't heard about it providing such safety. I would say this highly depends on what area you are coming from. In the cloud world it would be viewed way differently than in the dark^Wlow-level side of things. I don't want to compare libvirt to the kernel for example, but I haven't heard about C being a turn-off there. I'm sorry, but I think only handful of us (yeah, I think and hope I could count myself amoungst that group) are welcoming. But it's actually where I see one of the big turn-offs. It's hard to guess how that would turn out if C wasn't turn-off for them. Maybe we would get bunch of patchsets submitted that would be in better shape language-wise, but would that help with understanding various internal structures and behaviours of libvirt? Not considering the fact that some language would make it more readable. Or would we just get bunch more drive-by patches that we would have to fix/maintain? I don't think we can answer that. OK, and ... ... OK, I hope so and if you say so I will blindly believe that ;) Oh, good, I was hoping for this. But the code that is used by each binary will be present as many times as that binary is running since it is not dynamically loaded, right? Well, that sucks, but it's not a deal-breaker. Usually if you don't have SELinux and want to be guarded a tiny bit more. Like me. Maybe I'll cahnge that someday. I managed to fix that by having entry and exit scripts that mangle the TMPDIR env var for me. And it works in some cases. I have to read up on the basics on how to do a proper first-time setup for Go. And maybe everything will be sunshine and rainbows from that point forward. It's just that I don't like the fact that it's non-intuitive. Is it possible somehow to just build some code without actually dealing with bunch of dependencies and all the setup? To give an example, is there a way to differentiate the compiler from the dependency handling and build processes? Like gcc and autoconf/automake/make or rustc and cargo? Yeah. Good luck with evaluating the responses. I hope we won't need to change our Code of Conduct or even resort to voting...

On Mon, Nov 20, 2017 at 04:57:56PM +0000, Daniel P. Berrange wrote: Just a quick note on what I've found out after I dedicated half day to go through the tour of go and some other tutorials. The learning curve of Go is even less steep than I though (for some unknown reason) it is. So that's in favor of Go. However I haven't found out how is it possible to avoid some SIGSEGVs or aborts since Go doesn't have many recoverable errors. And in some cases they are not easy to spot immediately. Or making sure struct fields are initialized. Since libvirt strives to go for recoverable errors, I see this as a downside. Anyway, that's just to update others on what I've learnt. Have a nice day, Martin

On Tue, Nov 28, 2017 at 10:22:21AM +0000, Daniel P. Berrange wrote: Opps, meant to include this link https://blog.golang.org/defer-panic-and-recover Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Tue, Nov 28, 2017 at 9:44 AM Martin Kletzander <mkletzan(a)redhat.com&gt; wrote: Struct fields are always initialized to the zero value of the type, there is no such thing as uninitialized memory in go. https://golang.org/ref/spec#The_zero_value

On 11/20/2017 09:25 AM, Daniel P. Berrange wrote: To be fair, there's also the issue that getting a change into any external project and packaged into all the distros is more unpredictable (and may take longer) than implementing the same thing in their own project. RHEL (just as an example) has been updating libvirt roughly once a year for the past couple years. Chris

On Thu, Nov 23, 2017 at 11:32:26AM +0100, Michal Privoznik wrote: It depends on which areas we would want to mock/test. A regular Go method is not directly callable to from, and thus also not directly mockable with the ELF preload / overrides tricks do. A Go method which is explicitly exported with C calling convention, however, can be mocked in the same manner we do now. So if we had replaced some low level infrastructure with a Go module, we could mock at the public entry points to that Go module which were exported to C code. We could not mock the internals of that Go module. For pure Go code being tested in isolation from any C code, you would use a different approach. The general concept in Go is to define an 'interface' covering sets of APIs which you might want alternative impls for. So if there was a bit of Go code you wanted to easily replace in unit tests with a fake impl, you would define an interface and fake impl that satisifies that interface. So you need to put a little bit more thought into the code upfront to make it fakeable in tests, but if you do that the end result is comparable. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|