[libvirt] libvirt and the lowest common denominator
I wanted to bring this topic out of a patch review thread. Dan recently stated that only patches that use "upstream" facilities are acceptable, meaning the xend delivered by XenSource. I'd like to make a few points on this note: 1. I'd like to hear from the other core maintainers if they agree with this rule. 2. I'd like to hear a rationale for this rule. 3. The Sun repository *is* the upstream for xend on Solaris for a number of reasons: a. The 3.1.x series is only maintained by us, there is no other 3.1 upstream b. The XenSource upstream does not, and cannot, work on Solaris c. XenSource have no interest in xend at all and it's effectively unmaintained (with the exception of Novell I think). 4. I've seen stated a number of times that libvirt is designed to avoid becoming a lowest common denominator interface. I cannot reconcile that claim with this rule: all xend implementations must apparently be the same, and no enhancements are allowed. 5. This rule further restricts innovation in that XenSource must agree to any changes. This has a rather obvious chilling effect. The distro type/variant recording changes are a perfect example: this is essential information, and recording it in xend was pre-NAKed by XenSource a long time ago. 6. As a thought experiment: if someone re-wrote xend in C, what then? Is that upstream, or not? In case it's not clear, all of Sun's Xen changes are open source and easily available. 7. Any change that involves *incompatibility* is clearly different, and IMHO not acceptable. This is only about changes that in no way negatively affect behaviour with an upstream Linux xend. 8. I hope it's clear to everyone that I'm motivated to merge changes upstream wherever feasible. I'm not seeking some kind of get-out clause from being a responsible maintainer. 9. Practically, we will have to carry these changes in our libvirt no matter what. I'd rather not. 10. I would note that Red Hat own the upstream for the code they care about, thus this rule can only cause trouble for other people. My personal opinion, not Sun's. regards, john
Hi John, On Mon, 2009-02-09 at 10:39 -0500, John Levon wrote:
3. The Sun repository *is* the upstream for xend on Solaris for a number of reasons:
a. The 3.1.x series is only maintained by us, there is no other 3.1 upstream
b. The XenSource upstream does not, and cannot, work on Solaris
c. XenSource have no interest in xend at all and it's effectively unmaintained (with the exception of Novell I think). ... 7. Any change that involves *incompatibility* is clearly different, and IMHO not acceptable.
As a thought experiment - if upstream did gain the ability to work on Solaris, but it worked differently to Sun's version ... how would that be resolved in libvirt? Perhaps what you want is for libvirt to take the view that Sun's version of Xen is an incompatible fork and handle it differently, as if it was supporting a completely different virt platform? Cheers, Mark.
On Mon, Feb 09, 2009 at 04:03:31PM +0000, Mark McLoughlin wrote:
3. The Sun repository *is* the upstream for xend on Solaris for a number of reasons:
a. The 3.1.x series is only maintained by us, there is no other 3.1 upstream
b. The XenSource upstream does not, and cannot, work on Solaris
c. XenSource have no interest in xend at all and it's effectively unmaintained (with the exception of Novell I think). ... 7. Any change that involves *incompatibility* is clearly different, and IMHO not acceptable.
As a thought experiment - if upstream did gain the ability to work on Solaris, but it worked differently to Sun's version ... how would that be resolved in libvirt?
Are you talking about incompatible differences, or compatible ones? If it was incompatible, the only correct choice for libvirt project is to follow the community version. If it's compatible, then IMHO it's a case by case issue. The project would definitely follow the upstream version at least, but certain cases could support the Sun version too.
Perhaps what you want is for libvirt to take the view that Sun's version of Xen is an incompatible fork and handle it differently, as if it was supporting a completely different virt platform?
I would hope this isn't necessary. We've not yet had to deal with a case where there is one of these incompatible differences, and I personally don't intend to let it happen. This was in fact one of the main reasons for the big effort to merge up our libvirt changes! regards john
On Mon, Feb 09, 2009 at 10:39:55AM -0500, John Levon wrote:
I wanted to bring this topic out of a patch review thread.
Dan recently stated that only patches that use "upstream" facilities are acceptable, meaning the xend delivered by XenSource. I'd like to make a few points on this note:
1. I'd like to hear from the other core maintainers if they agree with this rule.
2. I'd like to hear a rationale for this rule.
The number one rule is not to fork codebases in incompatible ways. Adding new features to a particular version of Xen and not making any effort to send them back upstream is needlessly forking the code. This ripples into libvirt, because if upstream then adds the same feature but implements it in a different way, libvirt now has to satisfy two conflicting implementations, which hurts long term maintainability of our own code.
3. The Sun repository *is* the upstream for xend on Solaris for a number of reasons:
a. The 3.1.x series is only maintained by us, there is no other 3.1 upstream
b. The XenSource upstream does not, and cannot, work on Solaris
Are you in essence saying that long term you don't intend to follow the new Xen releases, and Solaris XenD should be considered a divergant fork long term ? Hopefully this isn't the case, but it'd mean we have to cope with the one libvirt Xen driver trying to serve & satisfy two different masters at once.
c. XenSource have no interest in xend at all and it's effectively unmaintained (with the exception of Novell I think).
While they may not have been doing much active new development work, they have still been accepting patches posted on xen-devel from a variety of sources, and doing new maintainence releases of the code.
4. I've seen stated a number of times that libvirt is designed to avoid becoming a lowest common denominator interface. I cannot reconcile that claim with this rule: all xend implementations must apparently be the same, and no enhancements are allowed.
This is not about lowest common denominator. It is about not needlessly forking code bases when adding new capabilities. The network device unplug enhancement is a perfect example. This is not Solaris specific in any way. If this were sent upstream and merged it serves to benefit all users of Xen, and means libvirt only has one implementation to worry about. If it is not sent upstream, and we put the Solaris speciifc code in place, and upstream then implements it in a different way, libvirt now has to carry 2 different impls of network unplug code. This is really not nice.
5. This rule further restricts innovation in that XenSource must agree to any changes. This has a rather obvious chilling effect. The distro type/variant recording changes are a perfect example: this is essential information, and recording it in xend was pre-NAKed by XenSource a long time ago.
IMHO, that's life in open source development. Sometimes things get nack'd and have to be implemented in a different manner to be acceptable to all members of the upstream community. Having everyone go off and do their own implementation of features when upstream reject them, is just not long term sustainable to the downstream users of that project being forked.
6. As a thought experiment: if someone re-wrote xend in C, what then? Is that upstream, or not? In case it's not clear, all of Sun's Xen changes are open source and easily available.
It would likely be a totally different product, requiring a from-scratch libvirt driver to support it. FWIW, we already have this scenario in the form of XenAPI + XenEnterprise server, which ditched current Xen RPC api that we use. If we added XenAPI support to libvirt, I expect it would be a completely new driver, whose code is independant of the existing libvirt Xen driver code. This would actually be an easier situation, because there'd be an explicit clean division between the implementations, and (within the confines of the libvirt API semantics & XML format) can evolve independantly of each other, not having to worry about one causing trouble for the other & vica-verca. The problem we have currently is harder two reconcile, because it involves trying to make 1 driver work with 2 potentially divergent projects.
7. Any change that involves *incompatibility* is clearly different, and IMHO not acceptable. This is only about changes that in no way negatively affect behaviour with an upstream Linux xend.
You need to consider future incompatability as well as any current incompatability. ie the scenario where you implement feature XYX one way, and the next upstream release implements it a completely different way. libvirt now has to carry 2 different implementations of the same code indefinitely, or choose to not support one of the forks. Neither is a good option really.
8. I hope it's clear to everyone that I'm motivated to merge changes upstream wherever feasible. I'm not seeking some kind of get-out clause from being a responsible maintainer.
9. Practically, we will have to carry these changes in our libvirt no matter what. I'd rather not.
10. I would note that Red Hat own the upstream for the code they care about, thus this rule can only cause trouble for other people.
Red Hat / Fedora always follow the rule of getting patches accepted by upstream Xen developers, despite fact that RHEL-5 is on ancient XenD 3.0.3, new features for 5.1, 5.2, 5.3, updates are all required to be upstream in xen-unstable first, and then backported. Likewise when we need new features for QEMU/KVM, we have to work with those development communities to get them to accept the required changes in QEMU codebae. On many occassions I've had code rejected outright, or rejected needing a re-write. The same rules for including new code in the QEMU driver of libvirt - if it needs a new QEMU command line argument, or monitor command, then that must be present in upstream QEMU/KVM codebase first. KVM is actually rather a PITA in some cases, because it has forked QEMU and added features, which then got re-implemented in QEMU. We got horribly burned by this in the case of migration whose command line syntax was changed. We should never have added support for that in libvirt while it was still in KVM only because the long term pain it caused outweighs the short term gain it had. NB, once a feature is upstream, then it *is* acceptable to backport to an existing release, because at that point you know that the patch will be short-lived until the next code rebase & is unlikely to be changed in an incompatible way. And downstream users of that feature (such as libvirt) only have one implementation they have to worry about. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Mon, Feb 09, 2009 at 04:54:51PM +0000, Daniel P. Berrange wrote:
2. I'd like to hear a rationale for this rule.
The number one rule is not to fork codebases in incompatible ways.
Agreed. Thankfully, we haven't done that.
Adding new features to a particular version of Xen and not making any effort to send them back upstream is needlessly forking the code.
A separate matter, but yes, efforts should be made to upstream things, I agree. I think if you look at the xen-devel history you'll see plenty of that.
3. The Sun repository *is* the upstream for xend on Solaris for a number of reasons:
a. The 3.1.x series is only maintained by us, there is no other 3.1 upstream
b. The XenSource upstream does not, and cannot, work on Solaris
Are you in essence saying that long term you don't intend to follow the new Xen releases, and Solaris XenD should be considered a divergant fork long term ?
Certainly not. However, moving to a new Xen version takes a very long time. The release quality is generally very poor, and combined with a Linux focus, it often takes weeks of work to move major versions. This is why we're still on 3.1.4. Practically, we *cannot* follow upstream directly. We will have enhancements; we'll have fixes; we'll disable completely broken code like the XenAPI implementation. This doesn't make us "divergent".
c. XenSource have no interest in xend at all and it's effectively unmaintained (with the exception of Novell I think).
While they may not have been doing much active new development work, they have still been accepting patches posted on xen-devel from a variety of sources, and doing new maintainence releases of the code.
This is very far away from what I consider maintenance.
forking code bases when adding new capabilities. The network device unplug enhancement is a perfect example. This is not Solaris specific in any way. If this were sent upstream and merged it serves to benefit all users of Xen
I'm not arguing an attempt shouldn't be made to upstream this particular patch. My points cover a wider issue.
worry about. If it is not sent upstream, and we put the Solaris speciifc code in place, and upstream then implements it in a different way, libvirt now has to carry 2 different impls of network unplug code. This is really not nice.
It wouldn't be ideal, thankfully I don't expect that to happen. But even if it did: 72 #ifdef WITH_RHEL5_API 73 #define XEND_CONFIG_MAX_VERS_NET_TYPE_IOEMU 0 74 #define XEND_CONFIG_MIN_VERS_PVFB_NEWCONF 2 75 #else 76 #define XEND_CONFIG_MAX_VERS_NET_TYPE_IOEMU 3 77 #define XEND_CONFIG_MIN_VERS_PVFB_NEWCONF 3 78 #endif This is exactly the same situation: you have a forked version of upstream that requires special handling.
5. This rule further restricts innovation in that XenSource must agree to any changes. This has a rather obvious chilling effect. The distro type/variant recording changes are a perfect example: this is essential information, and recording it in xend was pre-NAKed by XenSource a long time ago.
IMHO, that's life in open source development. Sometimes things get nack'd and have to be implemented in a different manner to be acceptable to all members of the upstream community.
There is no different manner without dropping xend altogether. You're perfectly happy for any stupid decision made by XenSource to negatively impact the libvirt project? Wow. There's no point having this argument if you're willing to throw away innovations. We won't stand for brokenness because of some bizarre policy, sorry.
6. As a thought experiment: if someone re-wrote xend in C, what then? Is that upstream, or not? In case it's not clear, all of Sun's Xen changes are open source and easily available.
It would likely be a totally different product, requiring a from-scratch libvirt driver to support it.
Let's see. Your policy has now led us to the situation where to enable storing of essential debug information, we have to duplicate several thousand lines of C code. john
On Mon, Feb 09, 2009 at 12:18:17PM -0500, John Levon wrote:
3. The Sun repository *is* the upstream for xend on Solaris for a number of reasons:
a. The 3.1.x series is only maintained by us, there is no other 3.1 upstream
b. The XenSource upstream does not, and cannot, work on Solaris
Are you in essence saying that long term you don't intend to follow the new Xen releases, and Solaris XenD should be considered a divergant fork long term ?
Certainly not. However, moving to a new Xen version takes a very long time. The release quality is generally very poor, and combined with a Linux focus, it often takes weeks of work to move major versions. This is why we're still on 3.1.4. Practically, we *cannot* follow upstream directly. We will have enhancements; we'll have fixes; we'll disable completely broken code like the XenAPI implementation. This doesn't make us "divergent".
Adding enhancements is what causes the problems & make it divergent, because the way you add a enhancement for feature XYZ is not guarenteed to be the way xen-unstable.hg does an enhancement for the same new feature XYZ. Bug fixes are fine, turning off unneccessary / broken bits is fine, but adding new features that are not first upstream causes problems.
c. XenSource have no interest in xend at all and it's effectively unmaintained (with the exception of Novell I think).
While they may not have been doing much active new development work, they have still been accepting patches posted on xen-devel from a variety of sources, and doing new maintainence releases of the code.
This is very far away from what I consider maintenance.
Likewise, but the fact remains that patches are posted & accepted into the codebase for issues like the ones being discused with device unplug of NICs.
worry about. If it is not sent upstream, and we put the Solaris speciifc code in place, and upstream then implements it in a different way, libvirt now has to carry 2 different impls of network unplug code. This is really not nice.
It wouldn't be ideal, thankfully I don't expect that to happen. But even if it did:
72 #ifdef WITH_RHEL5_API 73 #define XEND_CONFIG_MAX_VERS_NET_TYPE_IOEMU 0 74 #define XEND_CONFIG_MIN_VERS_PVFB_NEWCONF 2 75 #else 76 #define XEND_CONFIG_MAX_VERS_NET_TYPE_IOEMU 3 77 #define XEND_CONFIG_MIN_VERS_PVFB_NEWCONF 3 78 #endif
This is exactly the same situation: you have a forked version of upstream that requires special handling.
This is missing the point I have been trying to make. RHEL-5 is using the old Xen 3.0.3 codebase for XenD. We have added a large number of features to this over time. *ALL* of these features are things that are already available in upstream xen-unstable.hg tree. We do not add new features which are RHEL-5 only because that is not sustainable. This piece of code from libvirt you quote is not implementing any new features, it is merely turning on an existing feature. So that something that is upstream from 3.1.0 or later, is now also used when talking to Xen 3.0.4. This is why new features should be *backports* of existing stuff in upstream xen-unstable.hg - it means there only needs to be one functional implementation of the code in libvirt, which then only has to be turned on for corect version. Most of the features don't even need code like the lines you quoted above, becuse they 'just work' when backported from upstream.
5. This rule further restricts innovation in that XenSource must agree to any changes. This has a rather obvious chilling effect. The distro type/variant recording changes are a perfect example: this is essential information, and recording it in xend was pre-NAKed by XenSource a long time ago.
IMHO, that's life in open source development. Sometimes things get nack'd and have to be implemented in a different manner to be acceptable to all members of the upstream community.
There is no different manner without dropping xend altogether. You're perfectly happy for any stupid decision made by XenSource to negatively impact the libvirt project? Wow.
I didn't say I would be happy, just that things don't always work out as expected & sometimes need changing to meet the needs of the broader community, instead of just my our needs.
6. As a thought experiment: if someone re-wrote xend in C, what then? Is that upstream, or not? In case it's not clear, all of Sun's Xen changes are open source and easily available.
It would likely be a totally different product, requiring a from-scratch libvirt driver to support it.
Let's see. Your policy has now led us to the situation where to enable storing of essential debug information, we have to duplicate several thousand lines of C code.
Or work to get the neccessary feature upstream so it be used by libvirt and all other users of XenD without everyone having to re-invent it in their private code trees. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Tue, Feb 10, 2009 at 10:21:11AM +0000, Daniel P. Berrange wrote:
This is exactly the same situation: you have a forked version of upstream that requires special handling.
This is missing the point I have been trying to make. RHEL-5 is using the old Xen 3.0.3 codebase for XenD. We have added a large number of features to this over time. *ALL* of these features are things that are already
I don't think I am missing that point. I don't see much difference between the two cases, that's all.
It would likely be a totally different product, requiring a from-scratch libvirt driver to support it.
Let's see. Your policy has now led us to the situation where to enable storing of essential debug information, we have to duplicate several thousand lines of C code.
Or work to get the neccessary feature upstream so it be used by libvirt and all other users of XenD without everyone having to re-invent it in their private code trees.
Again, there is no work that can be done: it's forever NAKed. It's impossible to upstream it. It seems you are willing to let XenSource dictate what functionality libvirt is able to provide for Xen users. I care about improving the system even if XenSource's whims dictate otherwise, and it doesn't seem like we can reach agreement on this point. It's a pity our goals for what libvirt is all about aren't in agreement. regards, john
participants (3)
-
Daniel P. Berrange -
John Levon -
Mark McLoughlin