[libvirt] new openvz driver (bossonvz)
Hello, in the company I work for, we use openvz and qemu/kvm on our clusters side-by-side. To manage our domains, we used libvirt/qemu for qemu/kvm domains and vz tools for openvz domains in the past. This was very inconvinient since the management differed in many ways. So we have decided to unify our management and use libvirt exclusively. Since the openvz driver already included in libvirt lacks features that need, we have implemented a new libvirt backend driver for openvz called the bossonvz driver. Unlike the openvz driver, bossonvz is a complete, feature-rich stateful libvirt driver. It uses the libvirt driver API exclusively and communicates with the kernel directly, much like the LXC driver. The code is hugely inspired by the LXC driver and the Qemu driver, but adds a bit of functionality to the libvirt core too. More details and the source code can be found at http://bossonvz.bosson.eu/ The driver is completely independent of vz tools, it needs only a running vz kernel. One of the things, we are most proud of, is the possibility to access the domain's tty via VNC (hurray to uniform management web interfaces). Since the code was developed in-house (primarily for internal purposes), it is based on an older libvirt release (1.1.2). There are also some modifications to the libvirt core (virCommand) and the domain file (mount options syntax) which might need some redesign. At the moment I would like to get some feedback on the driver. In the future, I would like to see the driver merged upstream, and am prepared to do the work but I need to know if there is any interest in doing so. -- Cluster Design, s.r.o.
On Fri, Jun 27, 2014 at 03:16:51PM +0200, Bosson VZ wrote:
Hello,
Hello, first small rule here, trust is build on persons like most open source projects, so please change your email, I want to communicate with a person not with the name of a company.
in the company I work for, we use openvz and qemu/kvm on our clusters side-by-side. To manage our domains, we used libvirt/qemu for qemu/kvm domains and vz tools for openvz domains in the past. This was very inconvinient since the management differed in many ways. So we have decided to unify our management and use libvirt exclusively. Since the openvz driver already included in libvirt lacks features that need, we have implemented a new libvirt backend driver for openvz called the bossonvz driver.
it's true that the openvz diver is lagging behind a bit, I didn't see any real update since 2012. That wouldn't be the first time that we have multiple drivers for a given hypervisor, Xen is a precedent at least, though that's not a perfect situation as it dilutes manpower to maintain drivers for the hypervisor.
Unlike the openvz driver, bossonvz is a complete, feature-rich stateful libvirt driver. It uses the libvirt driver API exclusively and communicates with the kernel directly, much like the LXC driver. The code is hugely inspired by the LXC driver and the Qemu driver, but adds a bit of functionality to the libvirt core too. More details and the source code can be found at
The driver is completely independent of vz tools, it needs only a running vz kernel. One of the things, we are most proud of, is the possibility to access the domain's tty via VNC (hurray to uniform management web interfaces).
Since the code was developed in-house (primarily for internal purposes), it is based on an older libvirt release (1.1.2). There are also some modifications to the libvirt core (virCommand) and the domain file (mount options syntax) which might need some redesign.
At the moment I would like to get some feedback on the driver. In the future, I would like to see the driver merged upstream, and am prepared to do the work but I need to know if there is any interest in doing so.
The proper way to get feedback on the driver is: 1/ rebase your code to the current version, there is nothing we can do with a patch for 1.1.2 2/ split the changes logically, nobody will review a 44312 line patch, remove garbage from diffing generated files like configure 3/ submit the patches on this list for review, each patch having details about what it does, etc ... best manage all this with git on a branch from the upstream or you will be mad very quickly see how other does, that's standard practice and if you're not familiar with it, learning it won't be a lot time really. 4/ you will need to add patches adding documentation, no doc, no upstream :) If you think it's too much, then don't bother because we will need commitment to maintaining the driver over the years, and the sum of the work will be even larger than those initial steps. What you gain in exchange is integration in upstream, easier maintainance, easier rebase and recognition in the community. Makes sense ? Daniel -- Daniel Veillard | Open Source and Standards, Red Hat veillard@redhat.com | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | virtualization library http://libvirt.org/
Helo Daniel, thanks for the initial tips. I have submitted a new (more personal) email so within a few days all should be settled. To the feedback. I did not expect anything less than a request for a batch of smaller git patches. My first post here was merely to raise some public awareness of the driver and this seemed like a logical place where the information should go. I also wanted to know if I should even try to do the preparatory work for the patches considering there is already another OpenVZ driver included in libvirt. I will try to prepare some initial patches next week or so to test the proper way the code should flow upstream. -- David Fabian Cluster Desgin, s.r.o. Dne Po 30. června 2014 11:51:20, Daniel Veillard napsal(a):
On Fri, Jun 27, 2014 at 03:16:51PM +0200, Bosson VZ wrote:
Hello,
Hello,
first small rule here, trust is build on persons like most open source projects, so please change your email, I want to communicate with a person not with the name of a company.
in the company I work for, we use openvz and qemu/kvm on our clusters side-by-side. To manage our domains, we used libvirt/qemu for qemu/kvm domains and vz tools for openvz domains in the past. This was very inconvinient since the management differed in many ways. So we have decided to unify our management and use libvirt exclusively. Since the openvz driver already included in libvirt lacks features that need, we have implemented a new libvirt backend driver for openvz called the bossonvz driver.
it's true that the openvz diver is lagging behind a bit, I didn't see any real update since 2012. That wouldn't be the first time that we have multiple drivers for a given hypervisor, Xen is a precedent at least, though that's not a perfect situation as it dilutes manpower to maintain drivers for the hypervisor.
Unlike the openvz driver, bossonvz is a complete, feature-rich stateful libvirt driver. It uses the libvirt driver API exclusively and communicates with the kernel directly, much like the LXC driver. The code is hugely inspired by the LXC driver and the Qemu driver, but adds a bit of functionality to the libvirt core too. More details and the source code can be found at
The driver is completely independent of vz tools, it needs only a running vz kernel. One of the things, we are most proud of, is the possibility to access the domain's tty via VNC (hurray to uniform management web interfaces).
Since the code was developed in-house (primarily for internal purposes), it is based on an older libvirt release (1.1.2). There are also some modifications to the libvirt core (virCommand) and the domain file (mount options syntax) which might need some redesign.
At the moment I would like to get some feedback on the driver. In the future, I would like to see the driver merged upstream, and am prepared to do the work but I need to know if there is any interest in doing so.
The proper way to get feedback on the driver is: 1/ rebase your code to the current version, there is nothing we can do with a patch for 1.1.2 2/ split the changes logically, nobody will review a 44312 line patch, remove garbage from diffing generated files like configure 3/ submit the patches on this list for review, each patch having details about what it does, etc ... best manage all this with git on a branch from the upstream or you will be mad very quickly see how other does, that's standard practice and if you're not familiar with it, learning it won't be a lot time really. 4/ you will need to add patches adding documentation, no doc, no upstream :)
If you think it's too much, then don't bother because we will need commitment to maintaining the driver over the years, and the sum of the work will be even larger than those initial steps. What you gain in exchange is integration in upstream, easier maintainance, easier rebase and recognition in the community.
Makes sense ?
Daniel
Hi David, I think it's worth checking also if the technical approach used by thr driver will be compatible with current and future versions, not sure how much Parallels want to certify their current API/ABI kernel wise. Historically libvirt was started because we had similar issues with Xen at the time, so definitely something to check out to estimate the cost of maintaining the driver long term :) thanks, Daniel On Tue, Jul 01, 2014 at 02:37:49PM +0200, Bosson VZ wrote:
Helo Daniel,
thanks for the initial tips. I have submitted a new (more personal) email so within a few days all should be settled.
To the feedback. I did not expect anything less than a request for a batch of smaller git patches. My first post here was merely to raise some public awareness of the driver and this seemed like a logical place where the information should go. I also wanted to know if I should even try to do the preparatory work for the patches considering there is already another OpenVZ driver included in libvirt.
I will try to prepare some initial patches next week or so to test the proper way the code should flow upstream.
-- David Fabian Cluster Desgin, s.r.o.
Dne Po 30. června 2014 11:51:20, Daniel Veillard napsal(a):
On Fri, Jun 27, 2014 at 03:16:51PM +0200, Bosson VZ wrote:
Hello,
Hello,
first small rule here, trust is build on persons like most open source projects, so please change your email, I want to communicate with a person not with the name of a company.
in the company I work for, we use openvz and qemu/kvm on our clusters side-by-side. To manage our domains, we used libvirt/qemu for qemu/kvm domains and vz tools for openvz domains in the past. This was very inconvinient since the management differed in many ways. So we have decided to unify our management and use libvirt exclusively. Since the openvz driver already included in libvirt lacks features that need, we have implemented a new libvirt backend driver for openvz called the bossonvz driver.
it's true that the openvz diver is lagging behind a bit, I didn't see any real update since 2012. That wouldn't be the first time that we have multiple drivers for a given hypervisor, Xen is a precedent at least, though that's not a perfect situation as it dilutes manpower to maintain drivers for the hypervisor.
Unlike the openvz driver, bossonvz is a complete, feature-rich stateful libvirt driver. It uses the libvirt driver API exclusively and communicates with the kernel directly, much like the LXC driver. The code is hugely inspired by the LXC driver and the Qemu driver, but adds a bit of functionality to the libvirt core too. More details and the source code can be found at
The driver is completely independent of vz tools, it needs only a running vz kernel. One of the things, we are most proud of, is the possibility to access the domain's tty via VNC (hurray to uniform management web interfaces).
Since the code was developed in-house (primarily for internal purposes), it is based on an older libvirt release (1.1.2). There are also some modifications to the libvirt core (virCommand) and the domain file (mount options syntax) which might need some redesign.
At the moment I would like to get some feedback on the driver. In the future, I would like to see the driver merged upstream, and am prepared to do the work but I need to know if there is any interest in doing so.
The proper way to get feedback on the driver is: 1/ rebase your code to the current version, there is nothing we can do with a patch for 1.1.2 2/ split the changes logically, nobody will review a 44312 line patch, remove garbage from diffing generated files like configure 3/ submit the patches on this list for review, each patch having details about what it does, etc ... best manage all this with git on a branch from the upstream or you will be mad very quickly see how other does, that's standard practice and if you're not familiar with it, learning it won't be a lot time really. 4/ you will need to add patches adding documentation, no doc, no upstream :)
If you think it's too much, then don't bother because we will need commitment to maintaining the driver over the years, and the sum of the work will be even larger than those initial steps. What you gain in exchange is integration in upstream, easier maintainance, easier rebase and recognition in the community.
Makes sense ?
Daniel
-- Daniel Veillard | Open Source and Standards, Red Hat veillard@redhat.com | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | virtualization library http://libvirt.org/
On Fri, Jun 27, 2014 at 03:16:51PM +0200, Bosson VZ wrote:
Hello,
in the company I work for, we use openvz and qemu/kvm on our clusters side-by-side. To manage our domains, we used libvirt/qemu for qemu/kvm domains and vz tools for openvz domains in the past. This was very inconvinient since the management differed in many ways. So we have decided to unify our management and use libvirt exclusively. Since the openvz driver already included in libvirt lacks features that need, we have implemented a new libvirt backend driver for openvz called the bossonvz driver.
Are there any particular reasons why you were unable to extend the existing openvz driver to do what you needed ? I know openvz driver has been mostly unmaintained and its code isn't too pleasant, but it'd still be nice to know what show-stopper problems you saw with using it as a base for more work.
Unlike the openvz driver, bossonvz is a complete, feature-rich stateful libvirt driver. It uses the libvirt driver API exclusively and communicates with the kernel directly, much like the LXC driver. The code is hugely inspired by the LXC driver and the Qemu driver, but adds a bit of functionality to the libvirt core too. More details and the source code can be found at
The driver is completely independent of vz tools, it needs only a running vz kernel.
That's very interesting. Do you know if there is any statement of support for the OpenVZ kernel <-> userspace API. I know the mainline kernel aims to maintain userspace API compatibility, but are all the custom additions that the OpenVZ fork has maintained in a compatible manner as OpenVZ provides new versions ? eg is there any significant risk that when a new OpenVZ release comes out, kernel changes might break this new driver, or are they careful to maintain compat for the mgmt layer ? Also is there interoperability between this driver and the openvz tools. eg if openvz tools launch a guest, can it be seen and managed by this driver, and vica-verca ? Or are they completely separate views and non-interacting ?
One of the things, we are most proud of, is the possibility to access the domain's tty via VNC (hurray to uniform management web interfaces).
That's nice - care to elaborate on the technical architecture of this ? Presumably when you say 'tty' you mean the VNC is just a frontend to the text-mode console, it isn't providing any kind of virtualized graphical framebuffer. Is this VNC support you have part of the libvirt patches, or a standalone component that just interacts with it ? Basically I'm wondering whether this VNC support is something we can leverage in the LXC driver too ?
Since the code was developed in-house (primarily for internal purposes), it is based on an older libvirt release (1.1.2). There are also some modifications to the libvirt core (virCommand) and the domain file (mount options syntax) which might need some redesign.
At the moment I would like to get some feedback on the driver. In the future, I would like to see the driver merged upstream, and am prepared to do the work but I need to know if there is any interest in doing so.
As you've probably already figured out, we generally welcome support for new virtualization drivers. The two big questions to me are - Are you able to provide ongoing maintaince and development of this driver for the future ? We're already suffering from a lack of people able to maintain the existing openvz driver and the related parallels isn't too active either. Basically we'd like to see a commitment to be an active maintainer of the code. - What do you see as the long term future of the driver ? I've already asked whether there's any risk from future OpenVZ releases potentially breaking things. Historically the idea with the kernel's namespace support was that the OpenVZ forked kernel would be able to go away. IIUC, the openvz userspace can already run on top of a plain mainline kernel, albeit with reduced features compared to when it runs on openvz kernel fork. If we assume the trend of merging OpenVZ features into mainline continues, we might get to a point where OpenVZ kernel fork no longer exists. At that point I wonder what would be the compelling differences between BossonVZ and the LXC driver ? ie why would we benefit from 2 container drivers for the same kernel functionality. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Jun 30, 2014 at 10:31:26AM +0100, Daniel P. Berrange wrote:
On Fri, Jun 27, 2014 at 03:16:51PM +0200, Bosson VZ wrote: [...] - What do you see as the long term future of the driver ? I've already asked whether there's any risk from future OpenVZ releases potentially breaking things. Historically the idea with the kernel's namespace support was that the OpenVZ forked kernel would be able to go away. IIUC, the openvz userspace can already run on top of a plain mainline kernel, albeit with reduced features compared to when it runs on openvz kernel fork. If we assume the trend of merging OpenVZ features into mainline continues, we might get to a point where OpenVZ kernel fork no longer exists. At that point I wonder what would be the compelling differences between BossonVZ and the LXC driver ? ie why would we benefit from 2 container drivers for the same kernel functionality.
Good point ! My recollection of talking with James Bottomley is indeed that all the special features are being migrated as part of the commonly agreed upon set of kernel primitives put in the kernel for containers. So I hope that this driver will indeed work based on those primitives and not the Parallels specific extensions, as those may vanish sooner or later. What kernels do you support, only CentOS-6 modified ones ? Daniel -- Daniel Veillard | Open Source and Standards, Red Hat veillard@redhat.com | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | virtualization library http://libvirt.org/
Pavel and James to CC On 30/06/14 17:39, "Daniel Veillard" <veillard@redhat.com> wrote:
On Mon, Jun 30, 2014 at 10:31:26AM +0100, Daniel P. Berrange wrote:
On Fri, Jun 27, 2014 at 03:16:51PM +0200, Bosson VZ wrote: [...] - What do you see as the long term future of the driver ? I've already asked whether there's any risk from future OpenVZ releases potentially breaking things. Historically the idea with the kernel's namespace support was that the OpenVZ forked kernel would be able to go away. IIUC, the openvz userspace can already run on top of a plain mainline kernel, albeit with reduced features compared to when it runs on openvz kernel fork. If we assume the trend of merging OpenVZ features into mainline continues, we might get to a point where OpenVZ kernel fork no longer exists. At that point I wonder what would be the compelling differences between BossonVZ and the LXC driver ? ie why would we benefit from 2 container drivers for the same kernel functionality.
Good point !
My recollection of talking with James Bottomley is indeed that all the special features are being migrated as part of the commonly agreed upon set of kernel primitives put in the kernel for containers. So I hope that this driver will indeed work based on those primitives and not the Parallels specific extensions, as those may vanish sooner or later.
What kernels do you support, only CentOS-6 modified ones ?
Daniel
-- Daniel Veillard | Open Source and Standards, Red Hat veillard@redhat.com | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | virtualization library http://libvirt.org/
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 07/01/2014 07:43 PM, Dmitry Mishin wrote:
Pavel and James to CC
On 30/06/14 17:39, "Daniel Veillard" <veillard@redhat.com> wrote:
On Mon, Jun 30, 2014 at 10:31:26AM +0100, Daniel P. Berrange wrote:
On Fri, Jun 27, 2014 at 03:16:51PM +0200, Bosson VZ wrote: [...] - What do you see as the long term future of the driver ? I've already asked whether there's any risk from future OpenVZ releases potentially breaking things. Historically the idea with the kernel's namespace support was that the OpenVZ forked kernel would be able to go away. IIUC, the openvz userspace can already run on top of a plain mainline kernel, albeit with reduced features compared to when it runs on openvz kernel fork. If we assume the trend of merging OpenVZ features into mainline continues, we might get to a point where OpenVZ kernel fork no longer exists. At that point I wonder what would be the compelling differences between BossonVZ and the LXC driver ? ie why would we benefit from 2 container drivers for the same kernel functionality.
Good point !
My recollection of talking with James Bottomley is indeed that all the special features are being migrated as part of the commonly agreed upon set of kernel primitives put in the kernel for containers. So I hope that this driver will indeed work based on those primitives and not the Parallels specific extensions, as those may vanish sooner or later.
First of all -- great Thanks to the Bosson team for this work :) Unfortunately, I must say that our plan is to deprecate the OpenVZ kernel API for containers eventually. Current plan is the API for OpenVZ and PCS products would be the PCS SDK (the GPL-compatible version will be announced soon) and the libcontainer project. Thanks, Pavel
Thanks in return for the acknowledgement. I understand that the long-term plan is to reduce the size of the vzkernel patch and to only use vanilla kernel primitives Daniel was talking about. That would be an ideal situation. The thing is that we simply cannot wait for this merge to happen. We have been using OpenVZ for many years and bossonvz is just a convenient tool for us right now and, at least, in the mid-term future. My question is how is the new libcontainer/PCS SDK library going to influence the existing vz-tools eco-system? -- David Fabian Cluster Design, s.r.o. Dne Út 1. července 2014 19:52:28, Pavel Emelyanov napsal(a):
On 07/01/2014 07:43 PM, Dmitry Mishin wrote:
Pavel and James to CC
On 30/06/14 17:39, "Daniel Veillard" <veillard@redhat.com> wrote:
On Mon, Jun 30, 2014 at 10:31:26AM +0100, Daniel P. Berrange wrote:
On Fri, Jun 27, 2014 at 03:16:51PM +0200, Bosson VZ wrote: [...] - What do you see as the long term future of the driver ? I've already asked whether there's any risk from future OpenVZ releases potentially breaking things. Historically the idea with the kernel's namespace support was that the OpenVZ forked kernel would be able to go away. IIUC, the openvz userspace can already run on top of a plain mainline kernel, albeit with reduced features compared to when it runs on openvz kernel fork. If we assume the trend of merging OpenVZ features into mainline continues, we might get to a point where OpenVZ kernel fork no longer exists. At that point I wonder what would be the compelling differences between BossonVZ and the LXC driver ? ie why would we benefit from 2 container drivers for the same kernel functionality.
Good point !
My recollection of talking with James Bottomley is indeed that all the special features are being migrated as part of the commonly agreed upon set of kernel primitives put in the kernel for containers. So I hope that this driver will indeed work based on those primitives and not the Parallels specific extensions, as those may vanish sooner or later.
First of all -- great Thanks to the Bosson team for this work :)
Unfortunately, I must say that our plan is to deprecate the OpenVZ kernel API for containers eventually. Current plan is the API for OpenVZ and PCS products would be the PCS SDK (the GPL-compatible version will be announced soon) and the libcontainer project.
Thanks, Pavel
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 07/02/2014 04:15 PM, Bosson VZ wrote:
Thanks in return for the acknowledgement.
I understand that the long-term plan is to reduce the size of the vzkernel patch and to only use vanilla kernel primitives Daniel was talking about. That would be an ideal situation.
Yes, at the kernel level the long term plan is to eventually get rid of vz-specific API and switch to namespaces/cgroups completely. Thus any new users that sit on the vzkernel API make this process ... harder :)
The thing is that we simply cannot wait for this merge to happen. We have been using OpenVZ for many years and bossonvz is just a convenient tool for us right now and, at least, in the mid-term future. My question is how is the new libcontainer/PCS SDK library going to influence the existing vz-tools eco-system?
We will keep the CLI (vzctl) and library (libvzctl) working on new OpenVZ installations for some time (several years, I suppose) by making them properly redirect calls to new API. But all the new features will be implemented within the new API, libcontainer/PCS SDK. Thanks, Pavel
Fair enough. We have no problem to adapt to the new process. I don't know how libcontainer is designed internally and how low-level it is but having a thin user-space wrapper around the vzkernel would be wonderful. I wanted to use libvzctl in bossonvz to abstract the driver away from the low-level stuff but the library is just too much interconnected with the veconf parser. The current vz kernel API is not the nicest thing to work with to be honest and a shared user-space library would help both vzctl and bossonvz. If you are interested in my POV on the kernel API, I would say you should leave more work for the user-space. At the moment, when the user-space asks for a new container, the vzkernel magically creates a new environment and sets everything (esp. cgroups) by itself without any chance for the user-space to set anything. Because libvirt has its own cgroup hierarchy model and any libvirt driver should honor it, bossonvz should, idealy, create cgroups for the container under the libvirt hierarchy but it cannot since the kernel won't let it. As I see it, the driver has to do two separate things to successfully use the kernel API. There are the vz specific syscalls/ioctls (e.g. VZCTL_ENV_CREATE_DATA) and there is a preparatory work done in the user-space (preparing forked sub-processes, pipes, etc.). Is the new SDK going to cover only the low-level code (ioctls) or will I be able to use some sort of a high-level function like spawn_environment() and get PID of the newly created container's init? The amount of vz specific code in the bossonvz driver is not that big. The ioctls is just a single bossonvz_environment.c file. The harder part is the preparatory part which is located in bossonvz_container.c and is quite complex since it closely mimics the code path of vzctl (otherwise the driver would not work). -- David Fabian Cluster Design, s.r.o. Dne St 2. července 2014 19:41:15, Pavel Emelyanov napsal(a):
On 07/02/2014 04:15 PM, Bosson VZ wrote:
Thanks in return for the acknowledgement.
I understand that the long-term plan is to reduce the size of the vzkernel patch and to only use vanilla kernel primitives Daniel was talking about. That would be an ideal situation.
Yes, at the kernel level the long term plan is to eventually get rid of vz-specific API and switch to namespaces/cgroups completely. Thus any new users that sit on the vzkernel API make this process ... harder :)
The thing is that we simply cannot wait for this merge to happen. We have been using OpenVZ for many years and bossonvz is just a convenient tool for us right now and, at least, in the mid-term future. My question is how is the new libcontainer/PCS SDK library going to influence the existing vz-tools eco-system?
We will keep the CLI (vzctl) and library (libvzctl) working on new OpenVZ installations for some time (several years, I suppose) by making them properly redirect calls to new API.
But all the new features will be implemented within the new API, libcontainer/PCS SDK.
Thanks, Pavel
On 07/03/2014 03:49 PM, Bosson VZ wrote:
Fair enough. We have no problem to adapt to the new process. I don't know how libcontainer is designed internally and how low-level it is but having a thin user-space wrapper around the vzkernel would be wonderful.
Here's the preliminary version of it will look like: https://github.com/xemul/libct/ The API would get slightly shuffled to fit more the https://github.com/docker/libcontainer/pull/28
I wanted to use libvzctl in bossonvz to abstract the driver away from the low-level stuff but the library is just too much interconnected with the veconf parser.
Well, the libcontainer API is not going to mess with configs at all. Everything will be run-time.
The current vz kernel API is not the nicest thing to work with to be honest and a shared user-space library would help both vzctl and bossonvz. If you are interested in my POV on the kernel API, I would say you should leave more work for the user-space. At the moment, when the user-space asks for a new container, the vzkernel magically creates a new environment and sets everything (esp. cgroups) by itself without any chance for the user-space to set anything. Because libvirt has its own cgroup hierarchy model and any libvirt driver should honor it, bossonvz should, idealy, create cgroups for the container under the libvirt hierarchy but it cannot since the kernel won't let it.
Yes, this is our 15-years-old API that was kept compatible with tools. Soon after we started merging containers upstream we started the process of its deprecation, and in the upcoming Rhel7-based kernel we're very close to the goal.
As I see it, the driver has to do two separate things to successfully use the kernel API. There are the vz specific syscalls/ioctls (e.g. VZCTL_ENV_CREATE_DATA) and there is a preparatory work done in the user-space (preparing forked sub-processes, pipes, etc.). Is the new SDK going to cover only the low-level code (ioctls) or will I be able to use some sort of a high-level function like spawn_environment() and get PID of the newly created container's init?
The SDK is going to be quite high-level, and more sophisticated than existing libvzctl. The libcontainer is going to be low-level but still more abstract than the kernel API.
The amount of vz specific code in the bossonvz driver is not that big. The ioctls is just a single bossonvz_environment.c file. The harder part is the preparatory part which is located in bossonvz_container.c and is quite complex since it closely mimics the code path of vzctl (otherwise the driver would not work).
It's interesting. Can you point out where you code is? And, btw, I think you would be interested in taking part in libcontainer design, discussions and development. The link above is the place where we try to settle down the new API and you're welcome to join :) Thanks, Pavel
-- David Fabian Cluster Design, s.r.o. Dne Čt 3. července 2014 23:17:12, Pavel Emelyanov napsal(a):
On 07/03/2014 03:49 PM, Bosson VZ wrote:
Fair enough. We have no problem to adapt to the new process. I don't know how libcontainer is designed internally and how low-level it is but having a thin user-space wrapper around the vzkernel would be wonderful.
Here's the preliminary version of it will look like: https://github.com/xemul/libct/ The API would get slightly shuffled to fit more the https://github.com/docker/libcontainer/pull/28
I wanted to use libvzctl in bossonvz to abstract the driver away from the low-level stuff but the library is just too much interconnected with the veconf parser.
Well, the libcontainer API is not going to mess with configs at all. Everything will be run-time.
The current vz kernel API is not the nicest thing to work with to be honest and a shared user-space library would help both vzctl and bossonvz. If you are interested in my POV on the kernel API, I would say you should leave more work for the user-space. At the moment, when the user-space asks for a new container, the vzkernel magically creates a new environment and sets everything (esp. cgroups) by itself without any chance for the user-space to set anything. Because libvirt has its own cgroup hierarchy model and any libvirt driver should honor it, bossonvz should, idealy, create cgroups for the container under the libvirt hierarchy but it cannot since the kernel won't let it.
Yes, this is our 15-years-old API that was kept compatible with tools. Soon after we started merging containers upstream we started the process of its deprecation, and in the upcoming Rhel7-based kernel we're very close to the goal.
As I see it, the driver has to do two separate things to successfully use the kernel API. There are the vz specific syscalls/ioctls (e.g. VZCTL_ENV_CREATE_DATA) and there is a preparatory work done in the user-space (preparing forked sub-processes, pipes, etc.). Is the new SDK going to cover only the low-level code (ioctls) or will I be able to use some sort of a high-level function like spawn_environment() and get PID of the newly created container's init?
The SDK is going to be quite high-level, and more sophisticated than existing libvzctl. The libcontainer is going to be low-level but still more abstract than the kernel API.
The amount of vz specific code in the bossonvz driver is not that big. The ioctls is just a single bossonvz_environment.c file. The harder part is the preparatory part which is located in bossonvz_container.c and is quite complex since it closely mimics the code path of vzctl (otherwise the driver would not work).
It's interesting. Can you point out where you code is? And, btw, I think you would be interested in taking part in libcontainer design, discussions and development. The link above is the place where we try to settle down the new API and you're welcome to join :)
Thanks, Pavel
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Thanks for the links. I'll definitely check them out. The source code can be found at bossonvz.bosson.eu. But have in mind that the bossonvz driver uses only a subset of the vzkernel API. Some vzctl functionality is not implemented as it was not needed. Backwards compatibility with older vz kernels has also been dropped. And a lot of the auxiliary code is hidden in libvirt functions (especially virCommand). Thanks for the invitation. I believe the bossonvz driver could also leverage from that kind of co-operation. -- David Fabian Cluster Design, s.r.o. Dne Čt 3. července 2014 23:17:12, Pavel Emelyanov napsal(a):
On 07/03/2014 03:49 PM, Bosson VZ wrote:
Fair enough. We have no problem to adapt to the new process. I don't know how libcontainer is designed internally and how low-level it is but having a thin user-space wrapper around the vzkernel would be wonderful.
Here's the preliminary version of it will look like: https://github.com/xemul/libct/ The API would get slightly shuffled to fit more the https://github.com/docker/libcontainer/pull/28
I wanted to use libvzctl in bossonvz to abstract the driver away from the low-level stuff but the library is just too much interconnected with the veconf parser.
Well, the libcontainer API is not going to mess with configs at all. Everything will be run-time.
The current vz kernel API is not the nicest thing to work with to be honest and a shared user-space library would help both vzctl and bossonvz. If you are interested in my POV on the kernel API, I would say you should leave more work for the user-space. At the moment, when the user-space asks for a new container, the vzkernel magically creates a new environment and sets everything (esp. cgroups) by itself without any chance for the user-space to set anything. Because libvirt has its own cgroup hierarchy model and any libvirt driver should honor it, bossonvz should, idealy, create cgroups for the container under the libvirt hierarchy but it cannot since the kernel won't let it.
Yes, this is our 15-years-old API that was kept compatible with tools. Soon after we started merging containers upstream we started the process of its deprecation, and in the upcoming Rhel7-based kernel we're very close to the goal.
As I see it, the driver has to do two separate things to successfully use the kernel API. There are the vz specific syscalls/ioctls (e.g. VZCTL_ENV_CREATE_DATA) and there is a preparatory work done in the user-space (preparing forked sub-processes, pipes, etc.). Is the new SDK going to cover only the low-level code (ioctls) or will I be able to use some sort of a high-level function like spawn_environment() and get PID of the newly created container's init?
The SDK is going to be quite high-level, and more sophisticated than existing libvzctl. The libcontainer is going to be low-level but still more abstract than the kernel API.
The amount of vz specific code in the bossonvz driver is not that big. The ioctls is just a single bossonvz_environment.c file. The harder part is the preparatory part which is located in bossonvz_container.c and is quite complex since it closely mimics the code path of vzctl (otherwise the driver would not work).
It's interesting. Can you point out where you code is? And, btw, I think you would be interested in taking part in libcontainer design, discussions and development. The link above is the place where we try to settle down the new API and you're welcome to join :)
Thanks, Pavel
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Hello Daniel, I have started working on Bossonvz slightly more than a year ago. As far as I remember, back then I wanted to base my work on top of the existing Openvz libvirt driver but I was dissappointed by the state of the driver. Our goal was to have a driver that supports online migration and mount points management and after toying with the openvz driver for a while we have decided to start from the scratch. Personaly, I also do not like the idea of domain management by just calling the vz tools binaries and relying on parsing of the resulting outputs. The API stability is a very good question. Of course, to circumvent the vz tools and touching the kernel we had to accept the fact that the kernel API might change in the future. And it will most definitely do. This is not based on any formal proclamation in any way but it is true that in a past year I did not have to change my code because of some incompatibility in the vzkernel API (and I have tested the code with various vz kernels ranging from 042stab068.8 to 042stab084.26). At the moment, since the vz tools are the only user of the API, the vz developers can pretty much do whatever they like with the kernel API. I believe it could be benefitial also to them to have another user of their API. The implementations (vztools and bossonvz) are independent but they share the same kernel. This means that if I start a bossonvz domain, I can use vzlist and I will see the appropriate container. The opposite does not work that way because bossonvz is a stateful driver. So if I start a container by using vzctl, I will not see anything with virsh list. But I would not be able to start a domain with the same veid; libvirt won't let me do that because the kernel API call fails. By default, bossonvz also does not touch any vz tools config files (mainly veid.conf), so various vz tools that rely on the existence of the veid.conf for a running container (vzctl enter) will refuse to co-operate. This is solved by the bossonvz hook which can create stub veid.conf files to silence vzctl. After that, it is no problem to access the domain with vzctl enter (we use it regularly since entering the domain with virsh is not implemented yet). To sum up, the implementations influence one another to some extent but neither is necessary for the other and both can run simultaneuosly side-by-side. One of the goals we would like to reach is to have a uniform management interface for our clients. It means that the virtualization technology (openvz, qemu/kvm) should be abstracted away as much as possible. And because all qemu/libvirt management tools allow the user to see the "screen", we wanted the same for bossonvz. The implementation is located in bossonvz_vnc_* files. It is a stand-alone binary that is started as a separate process on the domain's startup by a domain controller (similar to the LXC domain controller). The VNC server binary opens up a tty device in the container via a vzkernel API call but instead of passing the IO through to the host pty device (like the LXC does), it actually intercepts the terminal sequences and renderes them to a memory frame-buffer (all in user-space). On top of that sits a custom VNC server implementation that serves the updated chunks to VNC clients. Everything is done in user-space, no modifications to the upstream vzkernel are necessary. The code heavily uses the libvirt util functions as well as uthash and some functions from libcrypto. Since the VNC server runs on the host, it is invisible to the user in the container. If for some reason the VNC server crashes, the domain controller can start it again without influencing the state of the domain itself. During migration (or a suspend/resume cycle), the state of the server is serialized to a file and stored on the domain's root partition. From there, it is deserialized and loaded when the domain is resumed. Since there is no dependency on the vzkernel API in the VNC server (apart from opening the tty device), there shall be no problem to use the code in the LXC driver. As far as the maintanence is concerned: Our bussinnes is centered around OpenVZ and Qemu/kvm technologies. Primarly, we are prepared to provide a long-term support of the bossonvz driver for every RedHat libvirt release and every stable vzkernel release because we use the driver already in production on RedHat based distros. Had the driver be accepted to the libvirt master with all the features that we need in the company, we are ready to provide support there as well. You are correct that LXC and OpenVZ cover the same thing, container virtualization. However, until LXC offers the same level of features as OpenVZ provides, we will use our bossonvz driver in the company. -- David Fabian Cluster Design, s.r.o. Dne Po 30. června 2014 10:31:26, Daniel P. Berrange napsal(a):
On Fri, Jun 27, 2014 at 03:16:51PM +0200, Bosson VZ wrote:
Hello,
in the company I work for, we use openvz and qemu/kvm on our clusters side-by-side. To manage our domains, we used libvirt/qemu for qemu/kvm domains and vz tools for openvz domains in the past. This was very inconvinient since the management differed in many ways. So we have decided to unify our management and use libvirt exclusively. Since the openvz driver already included in libvirt lacks features that need, we have implemented a new libvirt backend driver for openvz called the bossonvz driver.
Are there any particular reasons why you were unable to extend the existing openvz driver to do what you needed ? I know openvz driver has been mostly unmaintained and its code isn't too pleasant, but it'd still be nice to know what show-stopper problems you saw with using it as a base for more work.
Unlike the openvz driver, bossonvz is a complete, feature-rich stateful libvirt driver. It uses the libvirt driver API exclusively and communicates with the kernel directly, much like the LXC driver. The code is hugely inspired by the LXC driver and the Qemu driver, but adds a bit of functionality to the libvirt core too. More details and the source code can be found at
The driver is completely independent of vz tools, it needs only a running vz kernel.
That's very interesting. Do you know if there is any statement of support for the OpenVZ kernel <-> userspace API. I know the mainline kernel aims to maintain userspace API compatibility, but are all the custom additions that the OpenVZ fork has maintained in a compatible manner as OpenVZ provides new versions ?
eg is there any significant risk that when a new OpenVZ release comes out, kernel changes might break this new driver, or are they careful to maintain compat for the mgmt layer ?
Also is there interoperability between this driver and the openvz tools. eg if openvz tools launch a guest, can it be seen and managed by this driver, and vica-verca ? Or are they completely separate views and non-interacting ?
One of the things, we are most proud of, is the possibility to access the domain's tty via VNC (hurray to uniform management web interfaces).
That's nice - care to elaborate on the technical architecture of this ? Presumably when you say 'tty' you mean the VNC is just a frontend to the text-mode console, it isn't providing any kind of virtualized graphical framebuffer. Is this VNC support you have part of the libvirt patches, or a standalone component that just interacts with it ?
Basically I'm wondering whether this VNC support is something we can leverage in the LXC driver too ?
Since the code was developed in-house (primarily for internal purposes), it is based on an older libvirt release (1.1.2). There are also some modifications to the libvirt core (virCommand) and the domain file (mount options syntax) which might need some redesign.
At the moment I would like to get some feedback on the driver. In the future, I would like to see the driver merged upstream, and am prepared to do the work but I need to know if there is any interest in doing so.
As you've probably already figured out, we generally welcome support for new virtualization drivers. The two big questions to me are
- Are you able to provide ongoing maintaince and development of this driver for the future ? We're already suffering from a lack of people able to maintain the existing openvz driver and the related parallels isn't too active either. Basically we'd like to see a commitment to be an active maintainer of the code.
- What do you see as the long term future of the driver ? I've already asked whether there's any risk from future OpenVZ releases potentially breaking things. Historically the idea with the kernel's namespace support was that the OpenVZ forked kernel would be able to go away. IIUC, the openvz userspace can already run on top of a plain mainline kernel, albeit with reduced features compared to when it runs on openvz kernel fork. If we assume the trend of merging OpenVZ features into mainline continues, we might get to a point where OpenVZ kernel fork no longer exists. At that point I wonder what would be the compelling differences between BossonVZ and the LXC driver ? ie why would we benefit from 2 container drivers for the same kernel functionality.
Regards, Daniel
participants (5)
-
Bosson VZ -
Daniel P. Berrange -
Daniel Veillard -
Dmitry Mishin -
Pavel Emelyanov