9:25 a.m.
On Mon, Nov 20, 2017 at 12:24:22AM +0100, Martin Kletzander wrote:
On Tue, Nov 14, 2017 at 05:27:01PM +0000, Daniel P. Berrange wrote:
[...]
> I don't have direct experiance in Rust, but it has the same kind of benefits
over
> C as Go does, again without the downsides of languages like Python or Java. There
> are some interesting unique features to Rust that can be important to some apps.
> In particular it does not use garbage collection, instead the user must still do
> manual memory management as you would with C/C++. This allows Rust to be used in
> performance critical cases where it is unacceptable to have a garbage collector
> run. Despite a requirement for manual allocation/deallocation, Rust still
> provides a safe memory model. This approach of avoiding abstractions which will
> introduce performance overhead is a theme of Rust. The cost of such an approach
> is that development has a higher learning curve and ongoing cost in Rust, as
> compared to Go.
>
> I don't believe that the unique features of Rust, over Go, are important to the
> needs of libvirt. eg while for QEMU it would be critical to not have a GC
> doing asynchronous memory deallocation, this is not at all important to libvirt.
> In fact precisely the opposite, libvirt would benefit much more from having GC
> take care of deallocation, letting developers focus attention other areas. In
> general, as from having a memory safe language, what libvirt would most benefit
> from is productivity gains & ease of contribution. This is the core competancy
> of Go, and why it is the right choice for usage in libvirt.
>
So the first thing I disagreed on is that in Rust you do manual
allocations. In fact, you don't. Or, depending on the point of view,
you do less or the same amount of manual allocation than in Go. What is
the clear win for Rust it the concept of ownership and it's related to
the allocation mentioned before.
I shouldn't have used the word "allocation" in my paragraph above. As
you say, both languages have similar needs around allocation. The difference
I meant is around deallocation policy - in Rust, object lifetime is is a more
explicit decision on control of the progammer, as opposed to Go's garbage
collection. From what I've read Rust approach to deallocation is much
closer to the C++ concept of "smart pointers", eg this
http://pcwalton.github.io/blog/2013/03/18/an-overview-of-memory-managemen...
I am standing strongly behind the opinion that the learning curve of
Rust is definitely worth it. And coming from the C world, it is easy to
understand. To me, it is very easy to explain that concept to great
detail to someone who has background in libvirt. And the big benefit
(and still a huge opportunity for improvement WRT optimizations) is that
the compiler must know about it and so it is resolved compile-time.
Dereferencing or destructors are run at the end of their scope,
automatically. You can nicely see that when realizing that Rust doesn't
need any `defer` as Go has.
Nb the 'defer' concept isn't really about memory management per-se, rather
it is focused on cleanup of related resources - eg deciding when to close
an open file handle, or when to release another resource. Everything thats
just Go object memory is handling by the GC.
One of the things is that the kinks we have can be ironed out in C
as
well. It might be easier in other languages, but it is harder when you
have to switch to one. We have bunch of code dealing with backwards
compatibility. And I argue that this is something that causes issues on
its own. What's even worse, IMHO, is that we are so much feature-driven
that there is no time for any ironing. I see too much potential for
refactoring in various parts of libvirt that will never see the lights
of day because we need X to be implemented. And contributors sending
feature requests that they fail to maintain later don't help much with
that. Maybe we could fix this by saying the next Y releases will just
be bugfix releases. Maybe we could help bringing new contributors by
devoting some of our time to do an actual change that will make them
want to help us more. I know some of you will be sick and tired hearing
about Rust once more, but have you heard about how much their community
is inclusion-oriented? I guess what I'm trying to say is that there are
other (and maybe less disruptive) ways to handle the current problems we
are facing.
I'm not going to debate that there's plenty of problems we could be
tackling, and changing language is not a magic bullet for all of
them. My primary motivation is to start to get out of the world where
we have random crashes & security bugs due to overflowing buffers,
double frees, use after free, and so on. Problems that have been
solved by every language invented since C in the last 30-40 years.
On the choice of language, I will say C is a turn off to many people
as it is (not unreasonably) viewed as archaic, hard to learn and
difficult to write good code in.
When I worked in OpenStack it was a constant battle to get people to
consider enhancements to libvirt instead of reinventing it in Python.
It was a hard sell because most python dev just didn't want to use C
at all because it has a high curve to contributors, even if libvirt
as a community is welcoming. As a result OpenStack pretty much reinvented
its own hypervisor agnostic API for esx, hyperv, xenapi and KVM instead
of enhancing libvirt's support for esx, hyperv or xenapi. They only end
up using libvirt for KVM and libxl really. I hear similar comments from
people working in virt related projects in Go. So use of C really does
have an impact on our pool of potential contributors. This is disappointing
as there are huge numbers of people working on virt related projects, they
just don't ever go near C - virt-viewer & virsh are probably the only
"apps" using libvirt from C - all the others use a higher level language
via one of our bindings.
And then there are the "issues" with Go (and unfortunately
some with
Rust as well :'( ).
Yep, no choice is ever perfect.
Lot of the code for libraries is written with permissive licences,
but
if there is some that is LGPL-incompatible we can't use them. And in
ecosystems such as Rust and Go there are fewer alternatives, so we might
not find one that we'll be able to use. If that happens, there goes
bunch of our time. Like nothing.
I've not looked at the rust ecosystem, but from I've seen in Go most
devs tend to go for even more permissive licensing, ie BSD/Apache/MIT.
Disappointly few people pick GPL variants :-( So side from the
complication with our virtualbox code being GPLv2-only, I don't think
there's a license problem to worry about with Go, anymore than we
have to worry about with C today.
How do we deal with problems/bugs in dependence libraries? I know,
all
the projects are pretty new, so they might be nicer to contributors. If
they are not, we might need to fork or rewrite the code. Bam, another
chance of losing workforce.
Many C libs we depend on have been around along time so are more
mature, but are also more conservative in accepting changes,
especially if they touch API. On balance I don't think there would
be a big difference either way in this area.
How does Go handle updates in dependency libs? Does it
automatically
pull newest version from public repositories where some unknown person
can push whatever they want? Or can they be hash- or version-bound?
Originally it was quite informal (and awful) - your 3rd party deps
just had to be present in $GOPATH, and there was no tracking of
versions. No significant sized project works this way anymore
because it is insane.
Instead go introduced the concept they call the "vendoring" where
you have a top level dir called vendor/ where all your deps live.
You app provides a metadata file (in JSON typically) where you
list the deps and the preferred versions you need. The tool then
populates vendor/ with the right code to build against. Think
of vendor/ has been sort of like GIT submodules, but not using
GIT submodules, and you'll be thinking along the right lines.
Go libraries are being strongly encouraged to adopt semver for
their versioning.
The build process is that all the binaries are static, right? Or
all
the go code is static and it only has dynamic dependencies on C
libraries? In such a project as libvirt, wouldn't that mean that the
processes we run will be pretty heavy-weight?
Yes, all the Go code is statically linked - only C libs are dyn
loaded. This doesn't have any impact on runtime, because even if
the binary was 100's of MB in size, the kernel is only ever going
to page in sections of that file which are actually executed.
The main impact is that if a dependancy gets an update (eg for
a security fix) all downstream apps need rebuilding.
How is it with rebuilding after a small change. I know Go is good
when
it comes to compilation times. That might be something that people
might like a lot. Especially those who are trying to shave off every
second of compilation. However if you cannot use ccache and you always
need to rebuild everything, it might increase the build-time quite a
lot, even thought indirectly.
Compile times are great - the compiler is very fast, and it does
caching on a per-package basis. NB in go a "package" is an individual
directory in your source tree, so a typical app would have 10's
or 100's of packages, each corresponding to a separate subdir.
Source dirs are probably more fine grained that we use in libvirt.
eg what's in src/qemu in libvirt currently would likely end up
being spread across as many as 5 packages if it were idiomatic
Go.
The biggest win though comes from not needing autoconf, automake.
Of course libvirt wouldn't see that benefit as long as any of our
code were still C, so I won't claim that's a win in this particular
case.
You can't have /tmp mounted with noexec option unless you have
TMPDIR
set to some other directory. And sometimes not even in that case. I
guess non-issue for some distros, but bunch of people deal with that and
if seems like something that could be taken care of in Go itself and it
is just not.
I've not heard of that one before, and so obviously not hit it.
From Google it seems this applies if you use 'go run' or
'go test'
commands. The former is not something you typically use, but the
latter is. It can be avoided by having the make/shell run that
invokes 'go test' set a local TMPDIR - no need to set it globally
in your bash profile. I'm not sure what distros have /tmp with
noexec - Fedora doesn't at least.
You can't just clone a repo, cd into it and build it. You have
to get
the dependencies in a special manner, for that you have to have GOPATH
set and based on that you have to have your directories setup similarly
to what Go expects, then you need GOBIN if you want to build something
and other stuff that's just not nice and doesn't make much sense. At
least for newcomers. Simply the fact that it seems to me like Go is
trying to go against the philosophy of "Do one thing and do it well". I
know everyone is about having the build described in the same language
as the project, but what comes out of it is not something I prefer.
This is relataed to the thing I mention above where historically
everything was just splattered into $GOPATH. Most apps have gone
towards the vendoring concept where every dep is self-contained
in your local checkout.
I'm not against using another language to make some stuff better.
I
guess it is kind of visible from the mail that I like Rust, but I'm not
against other languages as well. I just want this to be full on
discussion and I want my opinion to be expressed. Thanks for
_listening_ ;)
Either Rust or Go would be a step forward over staying exclusively
with C IMHO, so at least we agree that there are potential benefits
to either :-)
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|