This is an initial stab at adding Linux-VServer support to libvirt. There are still a couple of things missing, like scheduler support in the XML parsing, and proper network support.

I've got a few questions though. virsh's schedinfo hardcodes the available options, should I be adding new ones there? Would better introspection from getSchedulerType make this a non-issue? How do the network domains interact and associate with the regular domains?

On Wed, Oct 31, 2007 at 04:09:54AM +0000, Daniel P. Berrange wrote: > On Tue, Oct 30, 2007 at 04:28:59PM +0100, Daniel Hokka Zakrisson wrote: >> This is an initial stab at adding Linux-VServer support to libvirt. >> There are still a couple of things missing, like scheduler support in >> the XML parsing, and proper network support. > Great to see interest in adding more drivers ! I've not had time to look Definitely, welcome aboard ! I'm just a bit worried that it's Yet Another Daniel in the project, confusion guaranteed :-)

> at your patch in great detail yet, but I'll give some more detailed > feedback in the next day or so. My first comment though - why the > <xid>123</xid> field in the XML - the unique integer ID for a domain > should really be in the 'id' attribute <domain id='123'>. There are a > couple of other small XML format consistency issues like that to check > up on. I looked at the code, that seems clean but I have a concern about the overall XML format. Could you paste a couple of examples. Also I think Linux-VServer and OpenVZ kind of configuration may end up with the same kind of limitations or differences, so I would like to try to harmonize both format when possible.

>> I've got a few questions though. virsh's schedinfo hardcodes the >> available options, should I be adding new ones there? Would better >> introspection from getSchedulerType make this a non-issue? How do the >> network domains interact and associate with the regular domains? > Yes, the virsh schedinfo command is broken in this respect. Rather > than hardcoding params, it should simply allow > > virsh schedinfo --set foo=bar --set wizz=123 > > To determine the data types for each param, it can simply query the existing > values with getSchedularInfo and then update them accordingly. the scheduling side of the API is probably the part where it's the harder to keep a coherent access between hypervisors, that's the reason why Rich designed a very flexible mechanism but drivers should use some introspection by looking at parameters names to process them, and provide good error reporting. virsh command can certainly be improved as Dan suggested, yes. thanks for the patch, but let's discuss and fix the XML format before trying to finish and push that first version :-) Daniel

On Wed, Oct 31, 2007 at 02:39:05PM +0100, Daniel Hokka Zakrisson wrote: > Daniel Veillard wrote: >> I looked at the code, that seems clean but I have a concern about the >> overall XML format. Could you paste a couple of examples. Also I think >> Linux-VServer and OpenVZ kind of configuration may end up with the same >> kind of limitations or differences, so I would like to try to harmonize >> both format when possible. > Currently, the XML format is really limited. Are there any docs on what > should be there, or should I just look at the other drivers? As far as > harmonizing with the OpenVZ driver, I'm fine with that, but it seems to > be pretty limited and, to some degree at least, ugly. Harmonizing the XML formats shouldn't be that hard ... We discussed the OpenVZ format there http://www.redhat.com/archives/libvir-list/2007-July/msg00347.html and around there earlier: http://www.redhat.com/archives/libvir-list/2007-March/msg00193.html For network settings http://www.redhat.com/archives/libvir-list/2007-July/msg00366.html

> Here's an example: > virsh # dumpxml etch > <domain type='vserver' id='40001'> > <name>etch</name> > <uuid>c81f40f2-7e72-606d-7d07-097c1d56a5b5</uuid> > <memory>500000</memory> > <scheduler> > <param name='fill_rate1'>100</param> > <param name='interval1'>1000</param> > <param name='fill_rate2'>25</param> > <param name='interval2'>1000</param> > <param name='idle_time'>1</param> > </scheduler> > </domain> I'm surprized there is no path or storage informations at all.

Are the parameters for the scheduler all integers ? If you really never end up with an information set more structured than that then <scheduler fill_rate1='100' interval1='1000' fill_rate2='25' interval2='1000' idle_time='1'/> would probably be simpler. The question is would other kind of scheduler use more structured parameters ? Seems to me it's not the case and that ad-hoc parsing to convert {name, value} pairs would just work.

Daniel

On Wed, Oct 31, 2007 at 05:14:10PM +0100, Daniel Hokka Zakrisson wrote: > Daniel Veillard wrote: > >Are the parameters for the scheduler all integers ? If you really > >never end up with an information set more structured than that then > > > > <scheduler fill_rate1='100' interval1='1000' fill_rate2='25' > > interval2='1000' > > idle_time='1'/> > > > >would probably be simpler. The question is would other kind of scheduler > >use more structured parameters ? Seems to me it's not the case and that > >ad-hoc parsing to convert {name, value} pairs would just work. > > Actually, the same settings can be applied to the CPUs separately, e.g. > <scheduler> > <param name='fill_rate1'>100</param> > <param name='interval1'>1000</param> > <cpu id='0'> > <param name='idle_time'>1</param> > <param name='fill_rate2'>25</param> > <param name='interval2'>1000</param> > </cpu> > </scheduler> > should at some point be possible (it's not in the current patch). Hum, I see, first time I see such fine grained scheduling, good to know, thanks ! I still find that <scheduler fill_rate1='100' interval1='1000'> <cpu id='0' idle_time='1' fill_rate2='25' interval2='1000'/> </scheduler> would be better than adding the arbitrary <param> element.

On Wed, Oct 31, 2007 at 02:39:05PM +0100, Daniel Hokka Zakrisson wrote: > Daniel Veillard wrote: >> I looked at the code, that seems clean but I have a concern about the >> overall XML format. Could you paste a couple of examples. Also I think >> Linux-VServer and OpenVZ kind of configuration may end up with the same >> kind of limitations or differences, so I would like to try to harmonize >> both format when possible. > Currently, the XML format is really limited. Are there any docs on what > should be there, or should I just look at the other drivers? As far as > harmonizing with the OpenVZ driver, I'm fine with that, but it seems to > be pretty limited and, to some degree at least, ugly. Harmonizing the XML formats shouldn't be that hard ... We discussed the OpenVZ format there http://www.redhat.com/archives/libvir-list/2007-July/msg00347.html and around there earlier: http://www.redhat.com/archives/libvir-list/2007-March/msg00193.html For network settings http://www.redhat.com/archives/libvir-list/2007-July/msg00366.html <snip>

On Thu, Nov 01, 2007 at 03:09:00AM +0100, Daniel Hokka Zakrisson wrote: > Daniel Veillard wrote: > >On Wed, Oct 31, 2007 at 02:39:05PM +0100, Daniel Hokka Zakrisson wrote: > >>Daniel Veillard wrote: > >>> I looked at the code, that seems clean but I have a concern about the > >>>overall XML format. Could you paste a couple of examples. Also I think > >>>Linux-VServer and OpenVZ kind of configuration may end up with the same > >>>kind of limitations or differences, so I would like to try to harmonize > >>>both format when possible. > >>Currently, the XML format is really limited. Are there any docs on what > >>should be there, or should I just look at the other drivers? As far as > >>harmonizing with the OpenVZ driver, I'm fine with that, but it seems to > >>be pretty limited and, to some degree at least, ugly. > > > > Harmonizing the XML formats shouldn't be that hard ... > >We discussed the OpenVZ format there > > http://www.redhat.com/archives/libvir-list/2007-July/msg00347.html > >and around there earlier: > > http://www.redhat.com/archives/libvir-list/2007-March/msg00193.html > >For network settings > > http://www.redhat.com/archives/libvir-list/2007-July/msg00366.html > > > > <snip> > > After looking at the various drivers, this is what I propose: > > <domain type='vserver' id='2000'> > <name>testvm</name> > <memory>500000</memory> > <scheduler fill_rate1='100' interval1='1000'> > <cpu id='0' fill_rate1='0'/> > <cpu id='1' fill_rate1='250' interval1='1000' idle_time='1' > fill_rate2='250' interval2='1000'/> > </scheduler> > <container> > <uts hostname='testvm' machine='i586'/> > <distribution method='debootstrap' release='etch'/> > </container> > <devices> > <disk type='path' device='path'> > <source path='/vservers/testvm'/> > <target path='/'/> > </disk> > <disk type='block' device='path'> > <source dev='/dev/hda4'/> > <target path='/var'/> > </disk> > <disk type='path' device='path'> > <source path='/home/testvm'/> > <target path='/home'/> > </disk> > <interface type='vserver'> > <ip family='IPv4' address='1.2.3.4' prefix='24' interface='eth0'/> > </interface> > <interface type='vserver'> > <ip family='IPv4' address='4.3.2.1' address2='4.3.2.200' > prefix='24' type='range'/> > </interface> > <interface type='vserver'> > <ip family='IPv6' address='dead:beef:dead:beef::1' prefix='64' > interface='eth0'/> > </interface> > </devices> > </domain> That looks mostly fine to me except a few doubts about: - <scheduler> this is tuning and we need a global discussion about this we allowed something similar for OpenVZ but this need to be discussed more globally. - what are the ranges for the values of <container> elements and attributes <uts> seems to be OS metadata, we already have an <os> element in existing configs why not reuse that. - some of the <ip> args are new, i.e. have no equivalent in existing config file uses: + family: that looks okay to me, is that mandatory ? Could that be guessed from one of the addresses given ? + address2: what does that mean ? I understand one IP address being served but since there is not really a notion of physical interface why attach 2 address instead of defining 2 <interface> ? + prefix: could you explain what this means ? if it's a netmask let's use something like netmask="255.255.255.0" instead of prefix="24" > I realize this is not at all similar to what the OpenVZ driver does, but > if necessary I could update that to use a similar scheme (though limited > in testing). -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard(a)redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ -- Libvir-list mailing list Libvir-list(a)redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

On Mon, Nov 05, 2007 at 07:57:36PM +0900, Atsushi SAKAI wrote: > Hi, Daniel > > Would you give me a pointer about discussion of OpenVZ (scheduler > parameter config)? > I searched the ML archive but I cannot find the discussion. some of it there http://www.redhat.com/archives/libvir-list/2007-July/msg00346.html for example > > > > Harmonizing the XML formats shouldn't be that hard ... > > > >We discussed the OpenVZ format there > > > > http://www.redhat.com/archives/libvir-list/2007-July/msg00347.html > > > >and around there earlier: > > > > http://www.redhat.com/archives/libvir-list/2007-March/msg00193.html > > > >For network settings > > > > http://www.redhat.com/archives/libvir-list/2007-July/msg00366.html I don't think we added scheduler informations to OpenVZ format in the end but clearly that was discussed. Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard(a)redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

On Mon, Nov 05, 2007 at 05:33:02AM -0500, Daniel Veillard wrote: > On Thu, Nov 01, 2007 at 03:09:00AM +0100, Daniel Hokka Zakrisson wrote: > > After looking at the various drivers, this is what I propose: > > > > <domain type='vserver' id='2000'> > > <name>testvm</name> > > <memory>500000</memory> > > <scheduler fill_rate1='100' interval1='1000'> > > <cpu id='0' fill_rate1='0'/> > > <cpu id='1' fill_rate1='250' interval1='1000' idle_time='1' > > fill_rate2='250' interval2='1000'/> > > </scheduler> > > <container> > > <uts hostname='testvm' machine='i586'/> > > <distribution method='debootstrap' release='etch'/> > > </container> > > <devices> > > <disk type='path' device='path'> > > <source path='/vservers/testvm'/> > > <target path='/'/> > > </disk> > > <disk type='block' device='path'> > > <source dev='/dev/hda4'/> > > <target path='/var'/> > > </disk> > > <disk type='path' device='path'> > > <source path='/home/testvm'/> > > <target path='/home'/> > > </disk> > > <interface type='vserver'> > > <ip family='IPv4' address='1.2.3.4' prefix='24' interface='eth0'/> > > </interface> > > <interface type='vserver'> > > <ip family='IPv4' address='4.3.2.1' address2='4.3.2.200' > > prefix='24' type='range'/> > > </interface> > > <interface type='vserver'> > > <ip family='IPv6' address='dead:beef:dead:beef::1' prefix='64' > > interface='eth0'/> > > </interface> > > </devices> > > </domain> > > That looks mostly fine to me except a few doubts about: > - <scheduler> this is tuning and we need a global discussion about this > we allowed something similar for OpenVZ but this need to be discussed > more globally. > - what are the ranges for the values of <container> elements and attributes > <uts> seems to be OS metadata, we already have an <os> element in > existing configs why not reuse that. > - some of the <ip> args are new, i.e. have no equivalent in existing > config file uses: > + family: that looks okay to me, is that mandatory ? Could that > be guessed from one of the addresses given ? Trivially - IPv4 is guarenteed to have a '.', and IPv6 is guarentted to have ':'.

> + address2: what does that mean ? I understand one IP address being > served but since there is not really a notion of physical interface > why attach 2 address instead of defining 2 <interface> ? I'd go for multiple <ip> tags inside the single <interface> tag, because this works in scenario where ythe 2nd address is a different family, or needs a different prefix

> + prefix: could you explain what this means ? if it's a netmask > let's use something like netmask="255.255.255.0" instead of > prefix="24" Prefix is now preferred nomenculture for what used to be called netmask. The use of netmask is impratical in IPv6 world. You can trivially convert between the two in an app if you need the legacy netmask. Basically, prefix refers to the number of bits in the address which are significant. eg, the number of bits set to 1, in the netmask.

I'd add that 'interface' should be in a separate <targe> element for consistency, and 'type=vserver' is probably redundant. We typically use just

type='ethernet' for a plain interface (ie one not bridged) <inteface type='ethernet'> <ip address="1.2.3.4" prefix='24'/> <target dev='eth0'/> </interface> <inteface type='ethernet'> <ip address="1.2.3.4" prefix='24'/> <ip address="dead:beef:dead:beed::1" prefix='48'/> <target dev='eth0'/> </interface>

Daniel Veillard wrote:

>> <container> >> <uts hostname='testvm' machine='i586'/> >> <distribution method='debootstrap' release='etch'/> >> </container>

> - what are the ranges for the values of <container> elements and > attributes I don't understand what this question means. > <uts> seems to be OS metadata, we already have an <os> element in > existing configs why not reuse that. Ah, yes, I missed that. How about: <os> <type machine='i586'>vserver</type> <hostname>testvm</hostname> <release>2.6.12</release> <distribution method='debootstrap'>etch</distribution> </os>

Daniel Veillard wrote: Ah, yes, I missed that. How about: <os> <type machine='i586'>vserver</type> <hostname>testvm</hostname> <release>2.6.12</release> <distribution method='debootstrap'>etch</distribution> </os>

On Tue, Oct 30, 2007 at 04:28:59PM +0100, Daniel Hokka Zakrisson wrote: > This is an initial stab at adding Linux-VServer support to libvirt. > There are still a couple of things missing, like scheduler support in > the XML parsing, and proper network support. Great to see interest in adding more drivers ! I've not had time to look at your patch in great detail yet, but I'll give some more detailed feedback in the next day or so. My first comment though - why the <xid>123</xid> field in the XML - the unique integer ID for a domain should really be in the 'id' attribute <domain id='123'>. There are a couple of other small XML format consistency issues like that to check up on.

NB, in most cases there is no need to implement network support in each driver. The libvirt networking APIs happen to be implemented in the QEMU driver codebase, but they're not really dependant on QEMU - the same functionality is shared across all drivers. If you connect to the Xen hypervisor, libvirt will auto-magically hook up the networking APIs to go via the QEMU driver. The same should work OK for VServer without you needing todo anything special.

> I've got a few questions though. virsh's schedinfo hardcodes the > available options, should I be adding new ones there? Would better > introspection from getSchedulerType make this a non-issue? How do the > network domains interact and associate with the regular domains? Yes, the virsh schedinfo command is broken in this respect. Rather than hardcoding params, it should simply allow virsh schedinfo --set foo=bar --set wizz=123 To determine the data types for each param, it can simply query the existing values with getSchedularInfo and then update them accordingly. Dan.

Daniel P. Berrange wrote: >On Tue, Oct 30, 2007 at 04:28:59PM +0100, Daniel Hokka Zakrisson wrote: >>This is an initial stab at adding Linux-VServer support to libvirt. >>There are still a couple of things missing, like scheduler support in >>the XML parsing, and proper network support. > >Great to see interest in adding more drivers ! I've not had time to look >at your patch in great detail yet, but I'll give some more detailed >feedback in the next day or so. My first comment though - why the ><xid>123</xid> field in the XML - the unique integer ID for a domain >should really be in the 'id' attribute <domain id='123'>. There are a >couple of other small XML format consistency issues like that to check >up on. Yeah, the only reason I did it with a separate element was that I really don't know XPath, so I hadn't figured out how to get the id in that case. >NB, in most cases there is no need to implement network support in each >driver. The libvirt networking APIs happen to be implemented in the QEMU >driver codebase, but they're not really dependant on QEMU - the same >functionality is shared across all drivers. If you connect to the Xen >hypervisor, libvirt will auto-magically hook up the networking APIs to >go via the QEMU driver. The same should work OK for VServer without >you needing todo anything special. Well, Linux-VServer is different from most (all?) other virtualization technologies in that we do IP isolation, rather than virtualizing the network stack. This means that guests are merely limited to a subset of the IP addresses assigned to the host, so there's no routing or bridging involved.