12:26 p.m.
Hi Everyone,
We are working towards building on the virtualization management
functionality in cockpit (http://cockpit-project.org/) and wanted to get
some feedback on the best way to integrate with libvirt.
As a quick overview, cockpit aims to talk to existing remotable system
APIs. Usually these API’s take the form of dbus, REST or executable
commands. The majority of cockpit is implemented in javascript. There is
no cockpit backend that knows how to change a hostname for example. The
cockpit backend knows how to handle a dbus payload. The javascript
running in the users browser knows how to use the systemd dbus API at
org.freedesktop.hostname1 to manage the system hostname.
Right now some of the basics have been implemented by spawning commands
on the system. This isn't ideal because it involves parsing / screen
scraping output and doesn't support receiving events so we have to poll
(ei run the command again) to keep the UI up to date.
As far as I know libvirt doesn't currently have a remoteable API. It
does have a daemon that communicates with clients via a XDR RPC.
(https://libvirt.org/internals/rpc.html) However from what I'm hearing
the RPC is considered an internal implementation and shouldn't be used
by external applications. Is that still the case? Is there any chance of
getting talking the daemon directly using the XDR standard for a subset
of methods blessed as part of the externally supported API?
An alternative is to implement a standards based remotable API, using
something like dbus or REST, that can be used by external applications.
I imagine that this would be at a bit of a higher level than the current
RPC and contain at least some of the logic around the actions it
performs rather than being a direct passthrough to the daemon.
Of course that is a pretty big undertaking and would, in my opinion,
only be worth it if there is broader interest in the community and use
cases beyond what cockpit would like to.
3:19 a.m.
On Thu, May 25, 2017 at 10:26:47AM -0700, Peter wrote:
As far as I know libvirt doesn't currently have a remoteable API.
It does
have a daemon that communicates with clients via a XDR RPC.
(https://libvirt.org/internals/rpc.html) However from what I'm hearing the
RPC is considered an internal implementation and shouldn't be used by
external applications. Is that still the case? Is there any chance of
getting talking the daemon directly using the XDR standard for a subset of
methods blessed as part of the externally supported API?
That's correct - the XDR protocol is a private implementation detail
between the libvirtd daemon and libvirt client library. The /only/
supported way of using libvirt is via the client library API, or one
of the many language bindings written on top of it. A number of the
libvirt virtualization drivers are implemented entirely in the client
library and don't involve libvirtd at all.
An alternative is to implement a standards based remotable API,
using
something like dbus or REST, that can be used by external applications.
I imagine that this would be at a bit of a higher level than the current RPC
and contain at least some of the logic around the actions it performs rather
than being a direct passthrough to the daemon.
Of course that is a pretty big undertaking and would, in my opinion, only be
worth it if there is broader interest in the community and use cases beyond
what cockpit would like to.
I imagine any DBus API would be a pretty straightforward 1-1 mapping from
the C api to DBus methods & signals. For a REST API I think you would quite
possibly want to build something higher level which will involve more design
thought.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
4:11 a.m.
On Thu, May 25, 2017 at 10:16:26AM -0700, Peter Volpe wrote:
Hi Everyone,
We are working towards building on the virtualization management
functionality in cockpit (http://cockpit-project.org/) and wanted to get
some feedback on the best way to integrate with libvirt.
As a quick overview, cockpit aims to talk to existing remotable system
APIs. Usually these API’s take the form of dbus, REST or executable
commands. The majority of cockpit is implemented in javascript. There is
no cockpit backend that knows how to change a hostname for example. The
cockpit backend knows how to handle a dbus payload. The javascript
running in the users browser knows how to use the systemd dbus API at
org.freedesktop.hostname1 to manage the system hostname.
Right now some of the basics have been implemented by spawning commands
on the system. This isn't ideal because it involves parsing / screen
scraping output and doesn't support receiving events so we have to poll
(ei run the command again) to keep the UI up to date.
As far as I know libvirt doesn't currently have a remoteable API. It
does have a daemon that communicates with clients via a XDR RPC.
(https://libvirt.org/internals/rpc.html) However from what I'm hearing
the RPC is considered an internal implementation and shouldn't be used
by external applications. Is that still the case? Is there any chance of
getting talking the daemon directly using the XDR standard for a subset
of methods blessed as part of the externally supported API?
If we standardize even the smallest part of the RPC, then it might screw
us immediately. We are keeping it private just so we can enhance the
APIs we already have. We don't know when we will need to change some
part of it.
An alternative is to implement a standards based remotable API, using
something like dbus or REST, that can be used by external applications.
I imagine that this would be at a bit of a higher level than the current
RPC and contain at least some of the logic around the actions it
performs rather than being a direct passthrough to the daemon.
Of course that is a pretty big undertaking and would, in my opinion,
only be worth it if there is broader interest in the community and use
cases beyond what cockpit would like to.
I was thinking of adding a REST API to libvirt for some time, actually.
It was supposed to be April fools patch because I didn't know about many
people who would actually like that. But it could've been made so that
there doesn't need to be much updated if new API arises. And there are
lot of things already implemented for JSON and the structures and types
we already use in libvirt. I never got around to doing that, though.
Nowadays, when thinking about this, there could be another layer of
abstraction between libvirt and consumers, and I understand REST would
fit a lot of them. Even though there would be some code duplication
from what already exists in libvirt, it might be worth it, I don't know.
I'm also not completely sure how much should be abstracted, but I feel
like not only cockpit, but oVirt and OpenStack (heck, maybe even
virt-manager) could benefit from such tool as well. Maybe all of you
could weigh in. I might, actually, be really interested in such a
project. That is if it shows that there is use for it (we all have
enough side projects that there is little to no use for them, right ;)),
so feel free to discuss the ideas to more detail.
Have a nice day,
Martin
P.S.: Since you started the discussion as the first one, I would suggest
paying homage to the cockpit project by naming it CockVirt.
9:45 a.m.
On 05/26/2017 02:11 AM, Martin Kletzander wrote:
On Thu, May 25, 2017 at 10:16:26AM -0700, Peter Volpe wrote:
> Hi Everyone,
>
> We are working towards building on the virtualization management
> functionality in cockpit (http://cockpit-project.org/) and wanted to get
> some feedback on the best way to integrate with libvirt.
>
> As a quick overview, cockpit aims to talk to existing remotable system
> APIs. Usually these API’s take the form of dbus, REST or executable
> commands. The majority of cockpit is implemented in javascript. There is
> no cockpit backend that knows how to change a hostname for example. The
> cockpit backend knows how to handle a dbus payload. The javascript
> running in the users browser knows how to use the systemd dbus API at
> org.freedesktop.hostname1 to manage the system hostname.
>
> Right now some of the basics have been implemented by spawning commands
> on the system. This isn't ideal because it involves parsing / screen
> scraping output and doesn't support receiving events so we have to poll
> (ei run the command again) to keep the UI up to date.
>
> As far as I know libvirt doesn't currently have a remoteable API. It
> does have a daemon that communicates with clients via a XDR RPC.
> (https://libvirt.org/internals/rpc.html) However from what I'm hearing
> the RPC is considered an internal implementation and shouldn't be used
> by external applications. Is that still the case? Is there any chance of
> getting talking the daemon directly using the XDR standard for a subset
> of methods blessed as part of the externally supported API?
>
If we standardize even the smallest part of the RPC, then it might screw
us immediately. We are keeping it private just so we can enhance the
APIs we already have. We don't know when we will need to change some
part of it.
How does the current ssh implementation work?
(https://libvirt.org/remote.html) If clients are able to talk to a
remote libvirtd via ssh then there must be some sort of compatibility
guarantee. Otherwise unless the versions are exactly the same clients
wouldn't be able to talk to remote daemons.
10:11 a.m.
On Tue, Jun 06, 2017 at 07:45:10 -0700, Peter wrote:
On 05/26/2017 02:11 AM, Martin Kletzander wrote:
> On Thu, May 25, 2017 at 10:16:26AM -0700, Peter Volpe wrote:
[...]
> If we standardize even the smallest part of the RPC, then it
might screw
> us immediately. We are keeping it private just so we can enhance the
> APIs we already have. We don't know when we will need to change some
> part of it.
>
How does the current ssh implementation work?
(https://libvirt.org/remote.html) If clients are able to talk to a remote
libvirtd via ssh then there must be some sort of compatibility guarantee.
Otherwise unless the versions are exactly the same clients wouldn't be able
to talk to remote daemons.
They use the internal RPC protocol transported over SSH. The client
library initializes the ssh connection to the server, and then starts to
talk the RPC tunelled over the SSH session.
The compatibility is guaranteed only if you use the client library. As
said, the RPC protocol is considered an internal detail and the client
library shields you from a possible incompatibility if we'd ever make
incompatible change.
10:17 a.m.
On Tue, Jun 06, 2017 at 05:11:55PM +0200, Peter Krempa wrote:
On Tue, Jun 06, 2017 at 07:45:10 -0700, Peter wrote:
> On 05/26/2017 02:11 AM, Martin Kletzander wrote:
> > On Thu, May 25, 2017 at 10:16:26AM -0700, Peter Volpe wrote:
[...]
> > If we standardize even the smallest part of the RPC, then it might screw
> > us immediately. We are keeping it private just so we can enhance the
> > APIs we already have. We don't know when we will need to change some
> > part of it.
> >
>
> How does the current ssh implementation work?
> (https://libvirt.org/remote.html) If clients are able to talk to a remote
> libvirtd via ssh then there must be some sort of compatibility guarantee.
> Otherwise unless the versions are exactly the same clients wouldn't be able
> to talk to remote daemons.
They use the internal RPC protocol transported over SSH. The client
library initializes the ssh connection to the server, and then starts to
talk the RPC tunelled over the SSH session.
The compatibility is guaranteed only if you use the client library. As
said, the RPC protocol is considered an internal detail and the client
library shields you from a possible incompatibility if we'd ever make
incompatible change.
Ignoring wire compatibility questions, it is important to note that not all
functionality in libvirt is done in libvirtd. There are libvirt drivers that
are entirely implemented in the library, not libvirtd. For some of the
libvirt drivers that do use libvirtd, there is also still some functionality
that is client side.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
11:07 a.m.
On 06.06.2017 17:17, Daniel P. Berrange wrote:
On Tue, Jun 06, 2017 at 05:11:55PM +0200, Peter Krempa wrote:
> On Tue, Jun 06, 2017 at 07:45:10 -0700, Peter wrote:
>> On 05/26/2017 02:11 AM, Martin Kletzander wrote:
>>> On Thu, May 25, 2017 at 10:16:26AM -0700, Peter Volpe wrote:
>
> [...]
>
>>> If we standardize even the smallest part of the RPC, then it might screw
>>> us immediately. We are keeping it private just so we can enhance the
>>> APIs we already have. We don't know when we will need to change some
>>> part of it.
>>>
>>
>> How does the current ssh implementation work?
>> (https://libvirt.org/remote.html) If clients are able to talk to a remote
>> libvirtd via ssh then there must be some sort of compatibility guarantee.
>> Otherwise unless the versions are exactly the same clients wouldn't be able
>> to talk to remote daemons.
>
> They use the internal RPC protocol transported over SSH. The client
> library initializes the ssh connection to the server, and then starts to
> talk the RPC tunelled over the SSH session.
>
> The compatibility is guaranteed only if you use the client library. As
> said, the RPC protocol is considered an internal detail and the client
> library shields you from a possible incompatibility if we'd ever make
> incompatible change.
Ignoring wire compatibility questions, it is important to note that not all
functionality in libvirt is done in libvirtd. There are libvirt drivers that
are entirely implemented in the library, not libvirtd. For some of the
libvirt drivers that do use libvirtd, there is also still some functionality
that is client side.
So to summarize what I'm hearing here:
* There is no libvirt remotable API today.
* All libvirt APIs are only able to be used in-process.
* There is presently no ability for a libvirt API to be used from a
browser, or non-Linux process or non-Linux system.
In order for Cockpit to interact with libvirt, it would need to grow a
remoteable API. The Cockpit project could likely jumpstart such an
effort, by lightly wrapping the C API in DBus ... and contribute that
remoteable API code to the libvirt project.
Cheers,
Stef
11:13 a.m.
On Tue, Jun 06, 2017 at 06:07:21PM +0200, Stef Walter wrote:
On 06.06.2017 17:17, Daniel P. Berrange wrote:
> On Tue, Jun 06, 2017 at 05:11:55PM +0200, Peter Krempa wrote:
>> On Tue, Jun 06, 2017 at 07:45:10 -0700, Peter wrote:
>>> On 05/26/2017 02:11 AM, Martin Kletzander wrote:
>>>> On Thu, May 25, 2017 at 10:16:26AM -0700, Peter Volpe wrote:
>>
>> [...]
>>
>>>> If we standardize even the smallest part of the RPC, then it might
screw
>>>> us immediately. We are keeping it private just so we can enhance the
>>>> APIs we already have. We don't know when we will need to change
some
>>>> part of it.
>>>>
>>>
>>> How does the current ssh implementation work?
>>> (https://libvirt.org/remote.html) If clients are able to talk to a remote
>>> libvirtd via ssh then there must be some sort of compatibility guarantee.
>>> Otherwise unless the versions are exactly the same clients wouldn't be
able
>>> to talk to remote daemons.
>>
>> They use the internal RPC protocol transported over SSH. The client
>> library initializes the ssh connection to the server, and then starts to
>> talk the RPC tunelled over the SSH session.
>>
>> The compatibility is guaranteed only if you use the client library. As
>> said, the RPC protocol is considered an internal detail and the client
>> library shields you from a possible incompatibility if we'd ever make
>> incompatible change.
>
> Ignoring wire compatibility questions, it is important to note that not all
> functionality in libvirt is done in libvirtd. There are libvirt drivers that
> are entirely implemented in the library, not libvirtd. For some of the
> libvirt drivers that do use libvirtd, there is also still some functionality
> that is client side.
So to summarize what I'm hearing here:
* There is no libvirt remotable API today.
Libvirt drivers that are implemented in libvirtd are accessible remotely
using the libvirt client library. We just consider the wire protocol a
private impl detail between them. Drivers implemented outside libvirtd
are often accessible remotely too, via the hypervisor's native remoting
system.
* All libvirt APIs are only able to be used in-process.
* There is presently no ability for a libvirt API to be used from a
browser, or non-Linux process or non-Linux system.
Actually libvirt can be built on OS-X, Windows and FreeBSD too.
In order for Cockpit to interact with libvirt, it would need to grow
a
remoteable API. The Cockpit project could likely jumpstart such an
effort, by lightly wrapping the C API in DBus ... and contribute that
remoteable API code to the libvirt project.
FWIW, if someone does want to build a DBus wrapper around libvirt, that
would be something todo as a separate code module above the libvirt
API. We would be happy to host such an effort on libvirt.org git though,
alongside other higher level API wrappers. Likewise for a REST API wrapper.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
12:25 p.m.
On 06.06.2017 12:13, Daniel P. Berrange wrote:
On Tue, Jun 06, 2017 at 06:07:21PM +0200, Stef Walter wrote:
> On 06.06.2017 17:17, Daniel P. Berrange wrote:
>> On Tue, Jun 06, 2017 at 05:11:55PM +0200, Peter Krempa wrote:
>>> On Tue, Jun 06, 2017 at 07:45:10 -0700, Peter wrote:
>>>> On 05/26/2017 02:11 AM, Martin Kletzander wrote:
>>>>> On Thu, May 25, 2017 at 10:16:26AM -0700, Peter Volpe wrote:
>>>
>>> [...]
>>>
>>>>> If we standardize even the smallest part of the RPC, then it might
screw
>>>>> us immediately. We are keeping it private just so we can enhance
the
>>>>> APIs we already have. We don't know when we will need to change
some
>>>>> part of it.
>>>>>
>>>>
>>>> How does the current ssh implementation work?
>>>> (https://libvirt.org/remote.html) If clients are able to talk to a
remote
>>>> libvirtd via ssh then there must be some sort of compatibility
guarantee.
>>>> Otherwise unless the versions are exactly the same clients wouldn't
be able
>>>> to talk to remote daemons.
>>>
>>> They use the internal RPC protocol transported over SSH. The client
>>> library initializes the ssh connection to the server, and then starts to
>>> talk the RPC tunelled over the SSH session.
>>>
>>> The compatibility is guaranteed only if you use the client library. As
>>> said, the RPC protocol is considered an internal detail and the client
>>> library shields you from a possible incompatibility if we'd ever make
>>> incompatible change.
>>
>> Ignoring wire compatibility questions, it is important to note that not all
>> functionality in libvirt is done in libvirtd. There are libvirt drivers that
>> are entirely implemented in the library, not libvirtd. For some of the
>> libvirt drivers that do use libvirtd, there is also still some functionality
>> that is client side.
>
> So to summarize what I'm hearing here:
>
> * There is no libvirt remotable API today.
Libvirt drivers that are implemented in libvirtd are accessible remotely
using the libvirt client library. We just consider the wire protocol a
private impl detail between them. Drivers implemented outside libvirtd
are often accessible remotely too, via the hypervisor's native remoting
system.
I believe those are implementation details, not and remoteable API.
Libvirt and its API must always be used in process today? Even though it
RPC may be in play, it is an implementation detail, not API. Correct?
> * All libvirt APIs are only able to be used in-process.
> * There is presently no ability for a libvirt API to be used from a
> browser, or non-Linux process or non-Linux system.
Actually libvirt can be built on OS-X, Windows and FreeBSD too.
Ah good to know. But the API is always used in process, correct?
> In order for Cockpit to interact with libvirt, it would need to
grow a
> remoteable API. The Cockpit project could likely jumpstart such an
> effort, by lightly wrapping the C API in DBus ... and contribute that
> remoteable API code to the libvirt project.
FWIW, if someone does want to build a DBus wrapper around libvirt, that
would be something todo as a separate code module above the libvirt
API. We would be happy to host such an effort on libvirt.org git though,
alongside other higher level API wrappers. Likewise for a REST API wrapper.
This would be a useful division of labor between Cockpit consuming such
a remotable API ... and the remoteable API being hosted and maintained
by the libvirt folks ... although the Cockpit project could help get it
started.
Stef
4:25 p.m.
On Tue, Jun 06, 2017 at 01:25:08PM -0400, Stef Walter wrote:
On 06.06.2017 12:13, Daniel P. Berrange wrote:
> On Tue, Jun 06, 2017 at 06:07:21PM +0200, Stef Walter wrote:
>> So to summarize what I'm hearing here:
>>
>> * There is no libvirt remotable API today.
>
> Libvirt drivers that are implemented in libvirtd are accessible remotely
> using the libvirt client library. We just consider the wire protocol a
> private impl detail between them. Drivers implemented outside libvirtd
> are often accessible remotely too, via the hypervisor's native remoting
> system.
I believe those are implementation details, not and remoteable API.
Libvirt and its API must always be used in process today? Even though it
RPC may be in play, it is an implementation detail, not API. Correct?
I don't really know what distinction your trying to make by saying
"used in process". Libvirt provides an ELF library and apps link to
it to use it, or use it indirectly via a higher level language specific
wrapper.
>> * All libvirt APIs are only able to be used in-process.
>> * There is presently no ability for a libvirt API to be used from a
>> browser, or non-Linux process or non-Linux system.
>
> Actually libvirt can be built on OS-X, Windows and FreeBSD too.
Ah good to know. But the API is always used in process, correct?
Again I don't know what you're trying to say here. It is an ELF
library and you link to it and call APIs as you would for any
other library.
>> In order for Cockpit to interact with libvirt, it would need
to grow a
>> remoteable API. The Cockpit project could likely jumpstart such an
>> effort, by lightly wrapping the C API in DBus ... and contribute that
>> remoteable API code to the libvirt project.
>
> FWIW, if someone does want to build a DBus wrapper around libvirt, that
> would be something todo as a separate code module above the libvirt
> API. We would be happy to host such an effort on libvirt.org git though,
> alongside other higher level API wrappers. Likewise for a REST API wrapper.
This would be a useful division of labor between Cockpit consuming such
a remotable API ... and the remoteable API being hosted and maintained
by the libvirt folks ... although the Cockpit project could help get it
started.
FWIW, in our RPC system, we auto-generate some of the marshalling code
from the API description XML file in /usr/share/libvirt/libvirt-api.xml
Some APIs are not friendly for auto-generating though, so probably get
about 75% auto-generated and the rest need to be done by hand.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
10:17 a.m.
On 06/06/2017 08:11 AM, Peter Krempa wrote:
On Tue, Jun 06, 2017 at 07:45:10 -0700, Peter wrote:
> On 05/26/2017 02:11 AM, Martin Kletzander wrote:
>> On Thu, May 25, 2017 at 10:16:26AM -0700, Peter Volpe wrote:
[...]
>> If we standardize even the smallest part of the RPC, then it might screw
>> us immediately. We are keeping it private just so we can enhance the
>> APIs we already have. We don't know when we will need to change some
>> part of it.
>>
>
> How does the current ssh implementation work?
> (https://libvirt.org/remote.html) If clients are able to talk to a remote
> libvirtd via ssh then there must be some sort of compatibility guarantee.
> Otherwise unless the versions are exactly the same clients wouldn't be able
> to talk to remote daemons.
They use the internal RPC protocol transported over SSH. The client
library initializes the ssh connection to the server, and then starts to
talk the RPC tunelled over the SSH session.
The compatibility is guaranteed only if you use the client library. As
said, the RPC protocol is considered an internal detail and the client
library shields you from a possible incompatibility if we'd ever make
incompatible change.
How does the client or server handle differing versions between them,
when the protocol changes? Or is it that client.x.x can only talk to
servers with the same exact version?
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
9:29 a.m.
On 06/06/2017 05:17 PM, Peter wrote:
On 06/06/2017 08:11 AM, Peter Krempa wrote:
> On Tue, Jun 06, 2017 at 07:45:10 -0700, Peter wrote:
>> On 05/26/2017 02:11 AM, Martin Kletzander wrote:
>>> On Thu, May 25, 2017 at 10:16:26AM -0700, Peter Volpe wrote:
>
> [...]
>
>>> If we standardize even the smallest part of the RPC, then it might
>>> screw
>>> us immediately. We are keeping it private just so we can enhance the
>>> APIs we already have. We don't know when we will need to change some
>>> part of it.
>>>
>>
>> How does the current ssh implementation work?
>> (https://libvirt.org/remote.html) If clients are able to talk to a
>> remote
>> libvirtd via ssh then there must be some sort of compatibility
>> guarantee.
>> Otherwise unless the versions are exactly the same clients wouldn't
>> be able
>> to talk to remote daemons.
>
> They use the internal RPC protocol transported over SSH. The client
> library initializes the ssh connection to the server, and then starts to
> talk the RPC tunelled over the SSH session.
>
> The compatibility is guaranteed only if you use the client library. As
> said, the RPC protocol is considered an internal detail and the client
> library shields you from a possible incompatibility if we'd ever make
> incompatible change.
>
How does the client or server handle differing versions between them,
when the protocol changes? Or is it that client.x.x can only talk to
servers with the same exact version?
So far there's just one version of the protocol. We try very hard not to
break it or not to introduce a new one. And frankly, we haven't really
had to so far.
But in general, we preserve backward compatibility therefore if we ever
introduce new version of the protocol, the old one is going nowhere so
that older clients can use it. But then again, this is highly hypothetical.
Michal
9:59 a.m.
On Thu, May 25, 2017 at 10:26:47AM -0700, Peter wrote:
The majority of cockpit is implemented in
javascript.
How about using the gobject libvirt bindings?
https://libvirt.org/git/?p=libvirt-glib.git;a=summary
They should be usable from Javascript directly, as in the .js example
here:
https://libvirt.org/git/?p=libvirt-glib.git;a=tree;f=examples;h=d63d5964b...
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
10:02 a.m.
On Wed, May 31, 2017 at 03:59:10PM +0100, Richard W.M. Jones wrote:
On Thu, May 25, 2017 at 10:26:47AM -0700, Peter wrote:
> The majority of cockpit is implemented in
> javascript.
How about using the gobject libvirt bindings?
https://libvirt.org/git/?p=libvirt-glib.git;a=summary
They should be usable from Javascript directly, as in the .js example
here:
https://libvirt.org/git/?p=libvirt-glib.git;a=tree;f=examples;h=d63d5964b...
They're usable from a standalone javascript interpretor, but there's no
way to use them from a client side javascript interpretor in the user's
web browser.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
10:52 a.m.
On Wed, May 31, 2017 at 04:02:42PM +0100, Daniel P. Berrange wrote:
On Wed, May 31, 2017 at 03:59:10PM +0100, Richard W.M. Jones wrote:
> On Thu, May 25, 2017 at 10:26:47AM -0700, Peter wrote:
> > The majority of cockpit is implemented in
> > javascript.
>
> How about using the gobject libvirt bindings?
>
> https://libvirt.org/git/?p=libvirt-glib.git;a=summary
>
> They should be usable from Javascript directly, as in the .js example
> here:
>
>
https://libvirt.org/git/?p=libvirt-glib.git;a=tree;f=examples;h=d63d5964b...
They're usable from a standalone javascript interpretor, but there's no
way to use them from a client side javascript interpretor in the user's
web browser.
Hmm OK, I thought "implemented in javascript" meant they were using
server-side Javascript.
I see that cockpit runs two processes on the server (cockpit-ws and
ssh-agent).
I had a look at one of the modules in the source (pkg/systemd). It's
running local commands (eg. "grep"), and issuing dbus calls which
AFAIK cannot be issued over the network.
Can Peter explain a bit more about the architecture of Cockpit?
Where does the Javascript run? What JS engine runs it?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
11:22 a.m.
On 05/31/2017 08:52 AM, Richard W.M. Jones wrote:
On Wed, May 31, 2017 at 04:02:42PM +0100, Daniel P. Berrange wrote:
> On Wed, May 31, 2017 at 03:59:10PM +0100, Richard W.M. Jones wrote:
>> On Thu, May 25, 2017 at 10:26:47AM -0700, Peter wrote:
>>> The majority of cockpit is implemented in
>>> javascript.
>>
>> How about using the gobject libvirt bindings?
>>
>> https://libvirt.org/git/?p=libvirt-glib.git;a=summary
>>
>> They should be usable from Javascript directly, as in the .js example
>> here:
>>
>>
https://libvirt.org/git/?p=libvirt-glib.git;a=tree;f=examples;h=d63d5964b...
>
> They're usable from a standalone javascript interpretor, but there's no
> way to use them from a client side javascript interpretor in the user's
> web browser.
Hmm OK, I thought "implemented in javascript" meant they were using
server-side Javascript.
I see that cockpit runs two processes on the server (cockpit-ws and
ssh-agent).
I had a look at one of the modules in the source (pkg/systemd). It's
running local commands (eg. "grep"), and issuing dbus calls which
AFAIK cannot be issued over the network.
Can Peter explain a bit more about the architecture of Cockpit?
Where does the Javascript run? What JS engine runs it?
Rich.
See this paragraph from my original message.
As a quick overview, cockpit aims to talk to existing remotable
system APIs. Usually these API’s take the form of dbus, REST or
executable commands. The majority of cockpit is implemented in
javascript. There is no cockpit backend that knows how to change a
hostname for example. The cockpit backend knows how to handle a dbus
payload. The javascript running in the users browser knows how to use
the systemd dbus API at org.freedesktop.hostname1 to manage the
system hostname.
The javascript is always run in the users browser. The dbus calls or
system commands are sent by the javascript via a websocket to
cockpit-ws. It then forwards those messages on to the correct
cockpit-bridge. Based on the payload the bridge knows how communicate
with the appropriate system API. (See
https://github.com/cockpit-project/cockpit/blob/master/doc/protocol.md
or run 'cockpit-bridge --version' for a list).
So in keeping with the cockpit ideals cockpit-bridge should not know how
to start a virtual machine or link to the libvirt library. It should
receive a payload that when properly executed by the bridge results in
the machine starting. Right now this is only possible using the virsh
type commands that have the problems discussed earlier. I think it's
clear that the existing RPC is not an option. So that pretty much leaves
us with the choice to roll or own solution or to add a more generic dbus
or rest wrapper for libvirt.
noon
On Wed, May 31, 2017 at 09:22:41AM -0700, Peter wrote:
The javascript is always run in the users browser. The dbus calls or
system commands are sent by the javascript via a websocket to
cockpit-ws. It then forwards those messages on to the correct
cockpit-bridge. Based on the payload the bridge knows how
communicate with the appropriate system API. (See
https://github.com/cockpit-project/cockpit/blob/master/doc/protocol.md
or run 'cockpit-bridge --version' for a list).
So in keeping with the cockpit ideals cockpit-bridge should not know
how to start a virtual machine or link to the libvirt library. It
should receive a payload that when properly executed by the bridge
results in the machine starting. Right now this is only possible
using the virsh type commands that have the problems discussed
earlier. I think it's clear that the existing RPC is not an option.
So that pretty much leaves us with the choice to roll or own
solution or to add a more generic dbus or rest wrapper for libvirt.
Thanks - that's a bit clearer now.
I agree with others that as things stand you will need a REST or DBus
or similar API added to libvirt.
However have you considered using gobject-introspection to generate
new "Payload" types automatically?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
1:46 p.m.
On 05/31/2017 01:00 PM, Richard W.M. Jones wrote:
On Wed, May 31, 2017 at 09:22:41AM -0700, Peter wrote:
> The javascript is always run in the users browser. The dbus calls or
> system commands are sent by the javascript via a websocket to
> cockpit-ws. It then forwards those messages on to the correct
> cockpit-bridge. Based on the payload the bridge knows how
> communicate with the appropriate system API. (See
https://github.com/cockpit-project/cockpit/blob/master/doc/protocol.md
> or run 'cockpit-bridge --version' for a list).
>
> So in keeping with the cockpit ideals cockpit-bridge should not know
> how to start a virtual machine or link to the libvirt library. It
> should receive a payload that when properly executed by the bridge
> results in the machine starting. Right now this is only possible
> using the virsh type commands that have the problems discussed
> earlier. I think it's clear that the existing RPC is not an option.
> So that pretty much leaves us with the choice to roll or own
> solution or to add a more generic dbus or rest wrapper for libvirt.
Thanks - that's a bit clearer now.
I agree with others that as things stand you will need a REST or DBus
or similar API added to libvirt.
One thing to consider though is that the libvirt API is only part of the
equation. The biggest part for sure, but if cockpit wants to create and manage
libvirt XML it should eventually use libvirt-gconfig, which is just a
standalone bit of code for manipulating the XML, and possibly libosinfo and
other libvirt-* libraries. Dan's writeup here describes the idea behind the
different libraries:
https://www.berrange.com/posts/2011/11/22/introducing-the-libvirt-glib-a-...
https://www.redhat.com/archives/libvir-list/2012-September/msg00325.html
Though the only ones that have had any real development live in
libvirt-glib.git: libvirt-glib, libvirt-gconfig, libvirt-gobject. And they
would likely need to be extended to fit cockpit's goals, but it's better than
writing all that XML building from scratch.
However have you considered using gobject-introspection to generate
new "Payload" types automatically?
This is an interesting idea (though I have no clue if it's technically
feasible). But if it worked it could be used to cover
libvirt-{gobject,glib,gconfig} and libosinfo which is likely everything
cockpit will ever need for virt management.
- Cole
12:49 a.m.
Hello Richard,
Richard W.M. Jones [2017-05-31 18:00 +0100]:
I agree with others that as things stand you will need a REST or
DBus
or similar API added to libvirt.
However have you considered using gobject-introspection to generate
new "Payload" types automatically?
This doesn't fundamentally change the picture here. Our JavaScript runs in the
browser, so on that side GI doesn't help. What we could do is to dynamically
create some code in the Cockpit module, send it over to the remote machine, and
have it be executed. This could be in Python, which then assumes that
python-libvirt is installed there, or in JS which then assumes that
libvirt-glib and gjs are installed. The latter seems like a stronger
assumption on a server even, but structurally these are pretty similiar.
I. e. GI is not a magic thing to make an API remotable, it just makes it
bindable to different programming languages.
C/GI interfaces also don't map well to D-Bus, i. e. it's not practical to
autogenerate a D-Bus interface for a given GI API. This still works for the
most simple methods that only accept primitive data types and strings, but as
soon as you pass any structs, pointers, function pointers etc. around this
can't be exposed though D-Bus any more without further interpretation on the
server side.
Martin
3:47 a.m.
On 07.06.2017 07:49, Martin Pitt wrote:
Hello Richard,
Richard W.M. Jones [2017-05-31 18:00 +0100]:
> I agree with others that as things stand you will need a REST or DBus
> or similar API added to libvirt.
>
> However have you considered using gobject-introspection to generate
> new "Payload" types automatically?
This doesn't fundamentally change the picture here. Our JavaScript runs in the
browser, so on that side GI doesn't help. What we could do is to dynamically
create some code in the Cockpit module, send it over to the remote machine, and
have it be executed. This could be in Python, which then assumes that
python-libvirt is installed there, or in JS which then assumes that
libvirt-glib and gjs are installed. The latter seems like a stronger
assumption on a server even, but structurally these are pretty similiar.
I. e. GI is not a magic thing to make an API remotable, it just makes it
bindable to different programming languages.
C/GI interfaces also don't map well to D-Bus, i. e. it's not practical to
autogenerate a D-Bus interface for a given GI API. This still works for the
most simple methods that only accept primitive data types and strings, but as
soon as you pass any structs, pointers, function pointers etc. around this
can't be exposed though D-Bus any more without further interpretation on the
server side.
Perhaps not an runtime. But it seems like such a thing could be done at
compile time for the libvirt C API. The libvirt folks do it for their
python bindings, using code generation.
The libvirt Python API behaves a lot like the C API. It's very flat and
not very "pythonic". Which is actually a nice target for us. We could do
something similar with a simple "flat" DBus wrapper of the API ... using
code generation.
What do you think?
Stef
3:55 a.m.
Stef Walter [2017-06-07 10:47 +0200]:
On 07.06.2017 07:49, Martin Pitt wrote:
> C/GI interfaces also don't map well to D-Bus, i. e. it's not practical to
> autogenerate a D-Bus interface for a given GI API. This still works for the
> most simple methods that only accept primitive data types and strings, but as
> soon as you pass any structs, pointers, function pointers etc. around this
> can't be exposed though D-Bus any more without further interpretation on the
> server side.
Perhaps not an runtime. But it seems like such a thing could be done at
compile time for the libvirt C API. The libvirt folks do it for their
python bindings, using code generation.
The libvirt Python API behaves a lot like the C API. It's very flat and
not very "pythonic". Which is actually a nice target for us. We could do
something similar with a simple "flat" DBus wrapper of the API ... using
code generation.
Sure, one can always write code to map the libvirt API into a D-Bus API (in C,
Python, or JavaScript with GJS, but I suppose for this use case Python might be
the best choice). My point was just that this would require to create,
package, and ship new code on the target machines, which is a very high barrier
for stable OS releases (particularly RHEL/CentOS, but it's not easy at all for
e. g. Debian or Ubuntu either).
Peter and I discussed if we could get away somehow with the existing code/APIs
on stable releases. E. g. a hackish, but maybe workable first step might be to
only use the python-libvirt API for subscribing to desired notifications and
trigger the Cockpit VM module through it, but continue to handle all the
actions through virsh. This subscription/notification code will hopefully be
small enough that it can be sent "through the wire" (cockpit channel) and
executed ad-hoc, without having to actually distribute it on the remote side.
Martin
_______________________________________________
cockpit-devel mailing list -- cockpit-devel(a)lists.fedorahosted.org
To unsubscribe send an email to cockpit-devel-leave(a)lists.fedorahosted.org
4:02 a.m.
On Wed, Jun 07, 2017 at 10:47:25AM +0200, Stef Walter wrote:
On 07.06.2017 07:49, Martin Pitt wrote:
> Hello Richard,
>
> Richard W.M. Jones [2017-05-31 18:00 +0100]:
>> I agree with others that as things stand you will need a REST or DBus
>> or similar API added to libvirt.
>>
>> However have you considered using gobject-introspection to generate
>> new "Payload" types automatically?
>
> This doesn't fundamentally change the picture here. Our JavaScript runs in the
> browser, so on that side GI doesn't help. What we could do is to dynamically
> create some code in the Cockpit module, send it over to the remote machine, and
> have it be executed. This could be in Python, which then assumes that
> python-libvirt is installed there, or in JS which then assumes that
> libvirt-glib and gjs are installed. The latter seems like a stronger
> assumption on a server even, but structurally these are pretty similiar.
>
> I. e. GI is not a magic thing to make an API remotable, it just makes it
> bindable to different programming languages.
>
> C/GI interfaces also don't map well to D-Bus, i. e. it's not practical to
> autogenerate a D-Bus interface for a given GI API. This still works for the
> most simple methods that only accept primitive data types and strings, but as
> soon as you pass any structs, pointers, function pointers etc. around this
> can't be exposed though D-Bus any more without further interpretation on the
> server side.
Perhaps not an runtime. But it seems like such a thing could be done at
compile time for the libvirt C API. The libvirt folks do it for their
python bindings, using code generation.
The libvirt Python API behaves a lot like the C API. It's very flat and
not very "pythonic". Which is actually a nice target for us. We could do
something similar with a simple "flat" DBus wrapper of the API ... using
code generation.
I see no problem with mapping almost all of the libvirt API into DBus, as
the DBus method call + signal concepts are very similar to the libvirt RPC
call and event concepts. The place where you would have trouble mapping
to DBus is the stream support as that has no equivalent in DBus, and there's
no efficient workaround to deal with it that I can imagine.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
4:06 a.m.
On 07.06.2017 11:02, Daniel P. Berrange wrote:
On Wed, Jun 07, 2017 at 10:47:25AM +0200, Stef Walter wrote:
> On 07.06.2017 07:49, Martin Pitt wrote:
>> Hello Richard,
>>
>> Richard W.M. Jones [2017-05-31 18:00 +0100]:
>>> I agree with others that as things stand you will need a REST or DBus
>>> or similar API added to libvirt.
>>>
>>> However have you considered using gobject-introspection to generate
>>> new "Payload" types automatically?
>>
>> This doesn't fundamentally change the picture here. Our JavaScript runs in
the
>> browser, so on that side GI doesn't help. What we could do is to dynamically
>> create some code in the Cockpit module, send it over to the remote machine, and
>> have it be executed. This could be in Python, which then assumes that
>> python-libvirt is installed there, or in JS which then assumes that
>> libvirt-glib and gjs are installed. The latter seems like a stronger
>> assumption on a server even, but structurally these are pretty similiar.
>>
>> I. e. GI is not a magic thing to make an API remotable, it just makes it
>> bindable to different programming languages.
>>
>> C/GI interfaces also don't map well to D-Bus, i. e. it's not practical
to
>> autogenerate a D-Bus interface for a given GI API. This still works for the
>> most simple methods that only accept primitive data types and strings, but as
>> soon as you pass any structs, pointers, function pointers etc. around this
>> can't be exposed though D-Bus any more without further interpretation on the
>> server side.
>
> Perhaps not an runtime. But it seems like such a thing could be done at
> compile time for the libvirt C API. The libvirt folks do it for their
> python bindings, using code generation.
>
> The libvirt Python API behaves a lot like the C API. It's very flat and
> not very "pythonic". Which is actually a nice target for us. We could do
> something similar with a simple "flat" DBus wrapper of the API ... using
> code generation.
I see no problem with mapping almost all of the libvirt API into DBus, as
the DBus method call + signal concepts are very similar to the libvirt RPC
call and event concepts. The place where you would have trouble mapping
to DBus is the stream support as that has no equivalent in DBus, and there's
no efficient workaround to deal with it that I can imagine.
Good to know.
FWIW, DBus supports FD passing. And Cockpit supports FD passing over DBus:
https://github.com/cockpit-project/cockpit/pull/5992
But I agree that any such wrapping of libvirt APIs involving streaming
in DBus FD passing would probably wouldn't be auto-generated.
Cheers,
Stef
3:48 a.m.
On Thu, May 25, 2017 at 10:26:47AM -0700, Peter wrote:
Hi Everyone,
We are working towards building on the virtualization management
functionality in cockpit (http://cockpit-project.org/) and wanted to get
some feedback on the best way to integrate with libvirt.
Another thing to consider is what virtualization management
functionality should be implemented in cockpit? For example if cockpit
would like to handle guest installation in a similar matter as
virt-manager it has to be implemented somewhere.
Libvirt doesn't have any API which would handle the whole installation
process and IMHO it's out of scope of dbus or REST API. The
installation process in virt-manager handles a lot of different aspects,
for example it can create a new storage/pool, it can download
vmlinuz/initrd from provided URL, it can mount an ISO image, etc.
This logic has to be implemented somewhere and I don't think that
cockpit is a right place. Currently there is command line tool
virt-install which is part of virt-manager project that is able to
handle all the installation aspects but that's not ideal for cockpit
the same way ho virsh is not ideal.
Another alternative way how to connect cockpit with libvirt could be to
implement some intermediate application which would communicate directly
with libvirt, would handle all the logic that is required by most of the
management application and provide several ways how UI applications can
communicate whit it. The benefit of this application would be that not
only cockpit would use it but virt-manager could switch to it eventually
and possibly virt-install as well. This would lead to a single project
that provides virtualization management functionality that can be
reused by other project or tools or simple scripts as well.
Pavel
As a quick overview, cockpit aims to talk to existing remotable
system
APIs. Usually these API’s take the form of dbus, REST or executable
commands. The majority of cockpit is implemented in javascript. There is
no cockpit backend that knows how to change a hostname for example. The
cockpit backend knows how to handle a dbus payload. The javascript
running in the users browser knows how to use the systemd dbus API at
org.freedesktop.hostname1 to manage the system hostname.
Right now some of the basics have been implemented by spawning commands
on the system. This isn't ideal because it involves parsing / screen
scraping output and doesn't support receiving events so we have to poll
(ei run the command again) to keep the UI up to date.
As far as I know libvirt doesn't currently have a remoteable API. It
does have a daemon that communicates with clients via a XDR RPC.
(https://libvirt.org/internals/rpc.html) However from what I'm hearing
the RPC is considered an internal implementation and shouldn't be used
by external applications. Is that still the case? Is there any chance of
getting talking the daemon directly using the XDR standard for a subset
of methods blessed as part of the externally supported API?
An alternative is to implement a standards based remotable API, using
something like dbus or REST, that can be used by external applications.
I imagine that this would be at a bit of a higher level than the current
RPC and contain at least some of the logic around the actions it
performs rather than being a direct passthrough to the daemon.
Of course that is a pretty big undertaking and would, in my opinion,
only be worth it if there is broader interest in the community and use
cases beyond what cockpit would like to.
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
2724
days inactive
2737
days old
23 comments
10 participants
participants (10)
-
Cole Robinson -
Daniel P. Berrange -
Martin Kletzander -
Martin Pitt -
Michal Privoznik -
Pavel Hrdina -
Peter -
Peter Krempa -
Richard W.M. Jones -
Stef Walter