Helo Daniel, thanks for the initial tips. I have submitted a new (more personal) email so within a few days all should be settled. To the feedback. I did not expect anything less than a request for a batch of smaller git patches. My first post here was merely to raise some public awareness of the driver and this seemed like a logical place where the information should go. I also wanted to know if I should even try to do the preparatory work for the patches considering there is already another OpenVZ driver included in libvirt. I will try to prepare some initial patches next week or so to test the proper way the code should flow upstream. -- David Fabian Cluster Desgin, s.r.o. Dne Po 30. června 2014 11:51:20, Daniel Veillard napsal(a):

On Fri, Jun 27, 2014 at 03:16:51PM +0200, Bosson VZ wrote: Are there any particular reasons why you were unable to extend the existing openvz driver to do what you needed ? I know openvz driver has been mostly unmaintained and its code isn't too pleasant, but it'd still be nice to know what show-stopper problems you saw with using it as a base for more work. That's very interesting. Do you know if there is any statement of support for the OpenVZ kernel <-> userspace API. I know the mainline kernel aims to maintain userspace API compatibility, but are all the custom additions that the OpenVZ fork has maintained in a compatible manner as OpenVZ provides new versions ? eg is there any significant risk that when a new OpenVZ release comes out, kernel changes might break this new driver, or are they careful to maintain compat for the mgmt layer ? Also is there interoperability between this driver and the openvz tools. eg if openvz tools launch a guest, can it be seen and managed by this driver, and vica-verca ? Or are they completely separate views and non-interacting ? That's nice - care to elaborate on the technical architecture of this ? Presumably when you say 'tty' you mean the VNC is just a frontend to the text-mode console, it isn't providing any kind of virtualized graphical framebuffer. Is this VNC support you have part of the libvirt patches, or a standalone component that just interacts with it ? Basically I'm wondering whether this VNC support is something we can leverage in the LXC driver too ? As you've probably already figured out, we generally welcome support for new virtualization drivers. The two big questions to me are - Are you able to provide ongoing maintaince and development of this driver for the future ? We're already suffering from a lack of people able to maintain the existing openvz driver and the related parallels isn't too active either. Basically we'd like to see a commitment to be an active maintainer of the code. - What do you see as the long term future of the driver ? I've already asked whether there's any risk from future OpenVZ releases potentially breaking things. Historically the idea with the kernel's namespace support was that the OpenVZ forked kernel would be able to go away. IIUC, the openvz userspace can already run on top of a plain mainline kernel, albeit with reduced features compared to when it runs on openvz kernel fork. If we assume the trend of merging OpenVZ features into mainline continues, we might get to a point where OpenVZ kernel fork no longer exists. At that point I wonder what would be the compelling differences between BossonVZ and the LXC driver ? ie why would we benefit from 2 container drivers for the same kernel functionality. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

Pavel and James to CC On 30/06/14 17:39, "Daniel Veillard" <veillard(a)redhat.com&gt; wrote:

Thanks in return for the acknowledgement. I understand that the long-term plan is to reduce the size of the vzkernel patch and to only use vanilla kernel primitives Daniel was talking about. That would be an ideal situation. The thing is that we simply cannot wait for this merge to happen. We have been using OpenVZ for many years and bossonvz is just a convenient tool for us right now and, at least, in the mid-term future. My question is how is the new libcontainer/PCS SDK library going to influence the existing vz-tools eco-system? -- David Fabian Cluster Design, s.r.o. Dne Út 1. července 2014 19:52:28, Pavel Emelyanov napsal(a):

Fair enough. We have no problem to adapt to the new process. I don't know how libcontainer is designed internally and how low-level it is but having a thin user-space wrapper around the vzkernel would be wonderful. I wanted to use libvzctl in bossonvz to abstract the driver away from the low-level stuff but the library is just too much interconnected with the veconf parser. The current vz kernel API is not the nicest thing to work with to be honest and a shared user-space library would help both vzctl and bossonvz. If you are interested in my POV on the kernel API, I would say you should leave more work for the user-space. At the moment, when the user-space asks for a new container, the vzkernel magically creates a new environment and sets everything (esp. cgroups) by itself without any chance for the user-space to set anything. Because libvirt has its own cgroup hierarchy model and any libvirt driver should honor it, bossonvz should, idealy, create cgroups for the container under the libvirt hierarchy but it cannot since the kernel won't let it. As I see it, the driver has to do two separate things to successfully use the kernel API. There are the vz specific syscalls/ioctls (e.g. VZCTL_ENV_CREATE_DATA) and there is a preparatory work done in the user-space (preparing forked sub-processes, pipes, etc.). Is the new SDK going to cover only the low-level code (ioctls) or will I be able to use some sort of a high-level function like spawn_environment() and get PID of the newly created container's init? The amount of vz specific code in the bossonvz driver is not that big. The ioctls is just a single bossonvz_environment.c file. The harder part is the preparatory part which is located in bossonvz_container.c and is quite complex since it closely mimics the code path of vzctl (otherwise the driver would not work). -- David Fabian Cluster Design, s.r.o. Dne St 2. července 2014 19:41:15, Pavel Emelyanov napsal(a):

-- David Fabian Cluster Design, s.r.o. Dne Čt 3. července 2014 23:17:12, Pavel Emelyanov napsal(a):

Hello Daniel, I have started working on Bossonvz slightly more than a year ago. As far as I remember, back then I wanted to base my work on top of the existing Openvz libvirt driver but I was dissappointed by the state of the driver. Our goal was to have a driver that supports online migration and mount points management and after toying with the openvz driver for a while we have decided to start from the scratch. Personaly, I also do not like the idea of domain management by just calling the vz tools binaries and relying on parsing of the resulting outputs. The API stability is a very good question. Of course, to circumvent the vz tools and touching the kernel we had to accept the fact that the kernel API might change in the future. And it will most definitely do. This is not based on any formal proclamation in any way but it is true that in a past year I did not have to change my code because of some incompatibility in the vzkernel API (and I have tested the code with various vz kernels ranging from 042stab068.8 to 042stab084.26). At the moment, since the vz tools are the only user of the API, the vz developers can pretty much do whatever they like with the kernel API. I believe it could be benefitial also to them to have another user of their API. The implementations (vztools and bossonvz) are independent but they share the same kernel. This means that if I start a bossonvz domain, I can use vzlist and I will see the appropriate container. The opposite does not work that way because bossonvz is a stateful driver. So if I start a container by using vzctl, I will not see anything with virsh list. But I would not be able to start a domain with the same veid; libvirt won't let me do that because the kernel API call fails. By default, bossonvz also does not touch any vz tools config files (mainly veid.conf), so various vz tools that rely on the existence of the veid.conf for a running container (vzctl enter) will refuse to co-operate. This is solved by the bossonvz hook which can create stub veid.conf files to silence vzctl. After that, it is no problem to access the domain with vzctl enter (we use it regularly since entering the domain with virsh is not implemented yet). To sum up, the implementations influence one another to some extent but neither is necessary for the other and both can run simultaneuosly side-by-side. One of the goals we would like to reach is to have a uniform management interface for our clients. It means that the virtualization technology (openvz, qemu/kvm) should be abstracted away as much as possible. And because all qemu/libvirt management tools allow the user to see the "screen", we wanted the same for bossonvz. The implementation is located in bossonvz_vnc_* files. It is a stand-alone binary that is started as a separate process on the domain's startup by a domain controller (similar to the LXC domain controller). The VNC server binary opens up a tty device in the container via a vzkernel API call but instead of passing the IO through to the host pty device (like the LXC does), it actually intercepts the terminal sequences and renderes them to a memory frame-buffer (all in user-space). On top of that sits a custom VNC server implementation that serves the updated chunks to VNC clients. Everything is done in user-space, no modifications to the upstream vzkernel are necessary. The code heavily uses the libvirt util functions as well as uthash and some functions from libcrypto. Since the VNC server runs on the host, it is invisible to the user in the container. If for some reason the VNC server crashes, the domain controller can start it again without influencing the state of the domain itself. During migration (or a suspend/resume cycle), the state of the server is serialized to a file and stored on the domain's root partition. From there, it is deserialized and loaded when the domain is resumed. Since there is no dependency on the vzkernel API in the VNC server (apart from opening the tty device), there shall be no problem to use the code in the LXC driver. As far as the maintanence is concerned: Our bussinnes is centered around OpenVZ and Qemu/kvm technologies. Primarly, we are prepared to provide a long-term support of the bossonvz driver for every RedHat libvirt release and every stable vzkernel release because we use the driver already in production on RedHat based distros. Had the driver be accepted to the libvirt master with all the features that we need in the company, we are ready to provide support there as well. You are correct that LXC and OpenVZ cover the same thing, container virtualization. However, until LXC offers the same level of features as OpenVZ provides, we will use our bossonvz driver in the company. -- David Fabian Cluster Design, s.r.o. Dne Po 30. června 2014 10:31:26, Daniel P. Berrange napsal(a):