On the need to move to a merge request workflow

We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so, but never made a concrete decision. I understand that some people will not like the idea of using merge requests. It is ok to feel that way, however, we need to consider the bigger picture and be open to learning to make the most of the opportunities, even if we don't like certain things about the tools. We need to recognise that there are millions of open source developers who are successfully delivering code using the merge request workflow, and we are just as capable as them. To claim that merge requests won't/can't work for libvirt, means there is something unique about the libvirt project's needs, and/or all those other developers are doing a worse job than libvirt developers, and/or libvirt developers are incapable of using tools that everyone else copes with. I don't believe any of those points are true. While it is certainly possible to come up with a list of points why we might like mailing lists, on aggregate merge requests will be the better solution. There are examples of high profile projects moving to a git forge based workflow and new projects use them as a first choice. If mailing lists were truly the better approach, then there would be a notable flow of projects moving away from merge requests, back to mailing lists. There is no evidence of a trend towards mailing list workflows. Short summary ============= This has quickly become another very long email to read though, so I'll attempt a summary * The need to attract new contributors is critical to the long term health of the project. Since 2016 the number of new contributors has dropped 50% from 80-90 per year, down to just 40-45 in 2019. * The current libvirt mailing lists are suffering from unreliable delivery. Some contributors are unable to send. Other contributors are unable to receive. * The contributors affected by the problems have very limited ability to resolve the problem they have, aside from opening tickets or changing email providers entirely. * A mail workflow relies on all contributors agreeing on the conventions to use when sending & replying to messages. Many of these conventions are unwritten and impose a burden for new contributors, discouraging their participation. * A mail workflow has many problems even for long term contributors, most notably with keeping track of what mails are outstanding. Each person has invented their own solution to deal with the lack of central tracking and they are all flawed & duplicating time on triage. * A merge request workflow has a lower barrier to participation for new contributors to the project, by making libvirt use commonly found workflow for OSS projects. * A merge request workflow is technically capable of supporting projects of equal size, or greater, than libvirt which deliver features at an equivalent speed. There is nothing special or unique about libvirts needs. See KDE, GNOME, SystemD, Kubernetes, OpenShift, to name just a few * Trying to directly match each aspect of a developer's current mail setup to a merge request workflow is not practical. They are inherantly different tools with differing capabilities which will never map 1:1 in all cases. * The largest barrier to a change is not technical, but the relative lack of experience & familiarity with the tools. This can only be solved by using them in a real world environment. A period of productivity disruption is inevitable while learning to optimize use of the new tools. * A merge request workflow is better for new or infrequent contributors, and in the long term is certainly usable for frequent contributors. Switching to merge requests is thus better on aggregate, and with experience/hindsight may well turn out better for everyone. With the last few points in mind in particular, the idea is to not do a big bang switchover of everything at once, but instead split it into phases. Initially the focus is to be on all the non-core projects, which means the language bindings, application integrations, and supporting tools. After those are all switched successfully, highlighting any areas of difficulty and allowing for some learning/adaptation, the main libvirt.git would be switched over. The non-core projects can be switched more or less immediately as their review demands are much lower. It is anticipated that the timeframe for everything would be on the order of 3-6 months, at the max. The long story ============== The measurement question ------------------------ The key motivation behind this proposal is that it will make libvirt a more attractive project for new contributors. The quotes later illustrate real world cases where our use of email workflow has hurt contributors. These are just the cases we have become aware of from people who were motivated enough to continue their effort despite the barriers faced. It is an unknown how many more people faced similar barriers and just walked away without us knowing. It has been suggested in the past that we run both workflows in parallel as a way to test which is better. I don't believe such an approach will provide valid data. Splitting attention between two different tools will be detrimental to the effectiveness of both tools. It would be an unbalanced comparison because there is no practical way to compare both tools with the same data set unless we wish to review all patch series twice. In addition the timeframes needed for a meaningful comparison would be too long, spanning 1+ years to gather statistically significant data on contribution. If a merge request workflow was a new, unproven concept, then it would none the less be worth trying to do a comparison, however, difficult. This is not the case though, merge request workflows have been around many years & proven viable. There is nothing special about the libvirt project or developers skills that would invalidate this track record. Thus this mail will look at various aspects of the tools and consider how they impact our work, with a particular focus on issues which negatively impact new contributors, or which have a discriminatory effect on certain subsets of new or existing contributors. Contributors decline -------------------- Since 2016 libvirt has seen a significant decline in the number of new authors contributing patches to the project. This is illustrated in the 3rd slide from this KVM Forum presentation: http://people.redhat.com/berrange/kvm-forum-2019/kvm-forum-2019-libvirt.pdf For the data behind this graph, the commits to all GIT repos on libvirt.git were analysed, taking the year recorded for the git changeset author date. The email addresses were recorded each year, and a "new contributor" was considered to be someone who had never made a commit in any previous calendar year. We can see the project grew quite fast until about 2011 and then levelled out with a reasonably consistent number of long term vs new contributors. Since 2016, however, there has been a clear declining trend. The raw data shows for the last two years, the non-core git repos in particular saw a big decline, however, the same is also seen for libvirt.git itself. There is likely no single factor responsible for this decline. A combination of interest in containers instead of VMs, and maturity of the codebase likely play a fairly large part. None the less, the decline of new contributors is highly undesirable, whatever the reason, as there is still plenty of work that needs doing in libvirt. Thus steps need to be taken to make libvirt more attractive to new contributors. Lowering the barrier to entry for contributors is one key aspect to this. New contributor mistakes ------------------------ Some of the mistakes new users are liable to make when dealing with patch submissions * HTML mail instead of plain text * Mangled patches due to mail client breaking long lines * Incorrectly threaded patch series * Not labelling series with version numbers * Sending plain ‘diff’ output instead of a git patch * Not basing the patch on recent git master * Not subscribing to the list before posting * Corporate legal privacy / copyright signatures * Unintelligible mail quoting in replies (Outlook) We've tried to steer people towards git-publish, as that has encoded many of the conventions to prevent mistakes, but that is by no means a foolproof solution. It still requires setup and a working mail server, as well as following git best practice wrt branch usage. New contributor real experiences -------------------------------- Some quotes on this theme that I've collected from just the last three months. I'm removing the names as this isn't about the individuals involved, rather their first / early impressions & experience of libvirt: "I have sent the patches to the list. I had seen a while back that eventually merge requests might be fielded, so I figured it was worth a shot." "just not familiar with manual patch work, normally been used to gitlab, github. I installed the git-publish as per document, but not sure about the next step from this tool. And, I am not sure about how to connect my email with this tool either." "I'm happy to hear you are considering a switch to gitlab. I've never contributed to projects which are using email based submission" "Have you guys received an email from .... with the subject ..... I can't find it in the archives" "I don't know what has happened, but we seem to be missing this patch from the list. ... I have all the patches in my inbox from the cc, so have no idea what happened either. It was my first time using git-publish and it took a few tries to get anything done." There are a variety of reasons behind these problems, but the effects are clear. Contributing to libvirt via email is complex and error prone. Mailman lost delivery --------------------- For Mailman, we are entirely dependent on Red Hat IT support via a private ticketing system. The reliability of Mailman leaves alot to be desired. For the past 5 years or more there have been multiple periods when mail delivery either stopped entirely, or randomly lost a subset of mails. When this happens if often isn't obvious at first that we're missing mail, and once identified, it has usually taken 2-3 days between opening a ticket and having someone actively investigate and resolve it. In the worst case it was over 7 days with broken mail delivery. This is not a good situation to be in, but we've tolerated it as nothing was irretrievably lost, just delayed by days, and every subscriber was equally affected. No service is perfectly reliable and we have seen both GitLab and GitHub have downtime periodically. A key difference though is how the problems are handled. For the git forges, service downtime is a business critical problem, and so gets the prioritized attention that it demands. Mailing list delivery for libvirt is NOT considered a business critical problem for Red Hat, and so the low priorization of fixing mailman problems reflects this. MX server content mangling -------------------------- More recently we've faced worse problems with the mailing list. The new service acting as redhat.com border MX gateway is mangling mails in a way that breaks DKIM signatures. Thus any time someone sends a message with a DKIM signature, there are a large number of subscribers to the list whose mail server will bounce the delivery from mailan due to DKIM validation failures. We tried to fix this by removing the subject prefix modification and body text footer in mailman, but that merely reduced the frequency of the problem by a small amount. In theory it can be mitigated by having mailman replace the "From" field with the list server's address. qemu-devel did this originally before they switched to removing the subject prefix/body footer. The reason it worked for QEMU was because only mailman was mangling their messages, whereas for libvirt, both mailman and the redhat.com MX gateway are mangling. The old version of mailman on redhat.com doesn't appear to support From header rewriting. Similar to DKIM problems, there are the same problems with ARC headers, though we lack definitive proof on the cause. The effect is that when any SUSE contributor sends a mail, deliveries to all other SUSE subscribers will be bounced. We believe it affects some other domains, but SUSE is the most obvious one. Both these problems are pretty serious and discriminate against certain subscribers depending on how their mail service configures their incoming filters. Even redhat.com was rejecting messages with broken DKIM signatures for a time, until enough people complained about missing valid emails. Many other contributors are not lucky enough to have an IT staff who would respond to such complaints. MX server anti-spam blackholing ------------------------------- A further problem has impacted mails sent by one contributor, which have been silently dropped by redhat.com MX servers, with no bounce at all. After two months we tracked this down to a bogus hostname in the From field of the messages which caused redhat.com to consider them a spammer spoofing the sender address & so blackhole it. This is actually a case where I probably agree with the redhat.com MX server policy, but it is none the less incredibly unhelpful for the unlucky contributor(s) affected by it, as it is hard for them to discover what they did wrong. We were lucky this contributor noticed they had a problem and was very persistent over months in talking to us about it to rather than walking away, as I expect most people would. Loosing members through attrition --------------------------------- Even without any of the above problems, the mailing list workflow still has the problem of being yet another account for people to keep track of. Every week I have to purge multiple subscribers from libvirt lists due to repeated bounced delivery attempts. Most commonly this is when their MX server starts rejecting messages claiming the address no longer exists (due to leaving a job). IOW, when someone has changed their job, and thus changed their email address, they have to re-register with any projects they are using. In some cases this is ok, as they may no longer care about the project after switching jobs, but it means we lack any channel to attempt to keep their interest. If another contributor tries to contact them via their old email address, they'll just get bounces too. With Git Forges, users have a login identity that is distinct from their email address, and thus they one need update one place when changing jobs, and interactions with other users are via this persistent identifier. Mail diagnosis / resolution --------------------------- The difficulty in all of these cases is that the project contributors who are affected have very limited means to either diagnose or resolve the problem. They are dependent on raising tickets with their own IT admin staff, who might then have to raise tickets with support from a company they outsourced to, and or talk to libvirt mailman admins. The libvirt mailman admins are similarly dependent on raising tickets with Red Hat IT and an external company. Depending on the level of outsourcing, the problem can span across IT support staff from 4/5 different companies. This has a predictably bad effect on the chances of resolving problems. Mail lack of recovery --------------------- Many of the problems result in permanent loss of messages to the subscriber affected. Even if the problem was eventually solved, there is often no straightforward way to get back the messages that were lost in the mean time. One the mail server has a hard bounce for that person, it will stop retrying delivery. The only option is to visit the mailman archive page and download the mbox and re-import messages. This is a result of email being a push based communication mechanism, with only a weak delivery assurance. If the push fails and isn't retried, the information is gone for good. Contrast with a Git Forge, where the primary communication mechanism is a pull based model. If there is a transient problem with the contributors network connectivity, or the hosted service has an outage, the user will be unable to pull information for a period of time. Once the problem is resolved, the next attempt to pull information will include everything, with no loss from the end user's POV. Mail list providers ------------------- Moving the libvirt mailing lists to another provider is not a magic bullet solution. qemu-devel is hosted on nongnu.org and has suffered bounces due to DKIM breakage, though at least their more modern mailman let them workaround it better. nongnu.org though has had similar problems with mail delivery getting stopped for periods of time, not as bad as libvirt suffered, but still dependant on waiting until the US wakes up to get anyone to investigate. I know other projects hosting lists on groups.io, which have also had serious problems of late with mails getting silently discarded, or mangled such that 'git am' fails. There are other providers The impact on contributors -------------------------- All this conspires to negatively impact the ability of people to participate in the libvirt project. If you can't reliably send & receive patches to libvir-list you are effectively excluded from the project, given that email is the core of our workflow. It hasn't had a huge visible impact on libvirt, primarily because the Red Hat contributions have escaped the worst effects. This "luck" doesn't make me at all happy, because it is important that anyone involved the project has an equal ability to contribute, regardless of whether they are Red Hat people or not. This is not the case There are many conceptual reasons to like a distributed messaging system based on open standards. The reality of the implementation of email though is quite different from the theory. The distributed nature of email, combined with the plethora of implementations which don't agree on what the minimum acceptable standards are, is the direct cause of the problems we have with reliable operation of the libvirt list, and mail in general. The problems with email don't seem to be getting better, if anything they are worse, as the broader world increasingly shifts away from email, sadly often to closed protocols, or closed applications. Overall though, I struggle the with idea of continuing to recommend email as the best choice for collaboration since it is clearly failing & excluding or harming some of our contributors. Mail usage for new contributors ------------------------------- For those of us who have been involved in libvirt for any length of time, the usage of email feels quite straightforward and natural. Our familiarity and experience can be quite different from that of new contribtors to the project. The usage of email as a development workflow relies on a large number of, often undocumented, conventions. If a contributor gets the conventions wrong they'll have poor first experience with the project. In the early days of libvirt those new contributors with existing open source experiance would already be familiar with email conventions, but we had plenty of contributors from the corporate world for which open source was a completely new experiance. Over the years we've sent nasty-gram responses to plenty of contributors who missed the conventions, though thankfully we're getting more polite and friendly these days in how we point out these kind of mistakes. It would be better to use a system that doesn't need this to the same degree. These days open source is a well known development model even among corporate developers from the closed source world, but their OSS knowledge is almost always based on a GitForge workflow, not an email workflow. Maintainer patch submission overload ------------------------------------ Any long term contributors to libvirt will have become familiar with the mail based development workflow, however, long term usage and familiarity does not imply that the solution used is the optional one, rather that the chosen solution has been optimized. With the volume of patches submitted to libvirt there is always pressure on reviewer bandwidth. The imbalance between incoming patches and reviewer time is common across all major open source projects, regardless of development workflow and tools. What matters is thus how well the project deals with the overload, to priortize the right aspects. The efficiency with which developers can perform reviews is one factor and is typically the area where email is said to be most beneficial. I personally find email is efficient for writing comments inline to patches, but that is largely due to the use of a text based user interface (mutt), rather than the inherent use of email as a messaging platform. There are other aspects to the use of email which harm our productivity as reviewers. Personal patch tracking ----------------------- To review a patch you first have to realize and/or remember that it exists, and this is an area where email is quite poor. It is easy for patches to go unattended for a prolonged period of time, drowned out in the volume of patches on the list. We encourage users to send a "ping" if they think their patch has been ignored for too long. This is a poor workaround for our inability to keep track of what patches are pending. New contributors won't even know they're expected to send a "ping", or may wonder how long they should wait. Essentially the mail folder needs to be thought of as a mixture of signal and noise, where 'signal' refers to patches which need reviewer attention, and 'noise' refers to patches which are already processed in some way. The most recent mails in an folder have a high signal component, turning increasingly to noise as they age. The key task for email processing is thus to distinguish signal from noise, such that we never miss any valid messages. The supposed positive of using email is that contributors have the flexibility to customize their way of working to enable them to keep track of pending work in whatever way suits them best. People come up with clever mail filtering practices / rules / scripts to organize their incoming mail stream. Instead of considering this a benefit, we should be asking why our tools are so ill-suited to the task that every contributor has to spend (waste) time reinventing the wheel to distinguish signal from noise when processing email, duplicating that same work across all contributors ? This has a particularly high burden on contributors who don't engage with the project on a daily basis, as they get a firehose of emails, many of which are already obsolete by the time they look at their inbox. IOW, in-frequent contributors experiance high noise, low signal, when interacting with the libvirt mailing list. At first glance merge requests may feel like they are no better, since there would be the same number of patches arriving each day. The key difference is that there is good structure and metadata to the merge requests. Multiple versions of the same series, appear as a single entity. Merge requests which have been merged or abandoned no longer appear. Active merge requests can be tagged with labels & users to organize them. Thus looking at the list of merge requests gives you information with a high signal content, and little noise content. This is good for both regular contributors and infrequent or new contributors. Patch tracking tools -------------------- With every contributor organizing their own personal incoming stream, there is no visibility / interaction between contributors on pending work. There is no way of knowing whether any given patch series is on the to do list for another person to deal with. There have been tools created which attempt to fill the gap by providing a centralized view of incoming patches. To name just a few, there are patchwork, patches, patchew, and patchecker. We have tried many of the tools to some degree or another with libvirt, but none have had success. Some are also in use in QEMU but see periodic breakage in consuming the mail stream for a variety of reasons. First and foremost, they suffer from the difficulty of turning free format mail threads into well structured data. This can only be done reliably if all contributors follow a set of well defined conventions for sending patches, which immediately puts a burden on contributors, and is a barrier to new contributors. These tools typically assume that all patches can apply against git master and if this isn't possible drop them with an error. Contributors rarely intentionally send patches against old versions of the code, but with the rate of commits, git master can still conflict with a newly posted patch series, requiring human intervention. Some support special magic comments in the mail to indicate what git version the patch is based on, but this is more secret sauce for contributors to learn. The problem of patch conflict resolution also creates challenges for the tools to determine when a series has been merged & can thus be considered complete. Then there is the matter of correlating different versions of the same patch series. This again largely relies on contributors following a convention of using the same subject line for the leading patch / cover letter each time it is posted. Some people post URLs pointing to previous versions of the same series, but this is yet more manual work to work around the limits of email when used for patch submission. Keeping a fixed subject does not always make sense. This again makes it hard for the tools to determine whether a patch series has been merged or obsoleted. So overall, while it is possible to write tools to turn incoming patch series emails into structured data, their reliability is poor, which in turn makes it difficult to consider relying on them. These same problems are why it is difficult to have a bi-directional email gateway between a mailing list and a merge request system. Sending mails from a merge request is easy. Parsing contextual replies and getting them back into the merge request is incredibly hard, often impossible. Attaching CI to submissions --------------------------- Without an ability to reliably process incoming patches in an automated manner, it is impractical to automate downstream tasks. It is generally considered best practice to run tests on all code before it is accepted for merge. Libvirt CI testing is currently all done post-merge. Contributors are generally expected to make sure code compiles and passes tests before posting it. The testing done this way rarely has the same breadth of coverage as that done by post-merge CI and may be done against an outdated git master. Reviewers may or may not take the time to actually apply the patches and run builds and tests. Where this is done, the coverage is usually just one or two platforms, a small subset of post-merge CI. The result of this is the frequent build & test breakage of git master, which has to be addressed through "brown paper bag" commits, mostly pushed without review under a "build breaker" rule. This is a prudent way to fix the breakage, however, it would be better if the breakage was prevented in the first place. If automation can catch the build problems before merge, there ceases to be any need for having special exceptions to allow pushing code without review. The result will be a higher quality output overall, with less time wasted on manually pointing out silly mistakes, or fixing them up after the fact. We know there is insufficient reviewer time to handle all the incoming patches submitted, and thus we should not expect them to spend time on tasks that can be automated. Reviewers should never be building code or running tests in general, or spending time pointing out problems in the code that can be caught by the tests. The tools should do all testing the moment the code is submitted, with reviewers only spending time once CI has passed & the submitted had a chance to re-upload any obvious fixes. The patchew tool attempts to do this kind of automation, but as noted above it suffers from the hard problem of reliably parsing email, determining what git master patches apply to, and needs secret comment syntax to be used by contributors for cases where it can't do the right thing. Barriers to new contributors ---------------------------- The only real solution to contributor overload is to spread the work across more people. This in turn implies a need to attract new contributors to the project. There need to be interesting problems to attract their attention. The task is then need to turn them into regular contributors, and this is where first impressions of a project are important. Simply put, the first interaction needs to be a positive experience. This might be via a bug tracker or via a code submission, or something else. Anything which may hurt first impressions needs to be eliminate or mitigated as much as practical. A need to learn unusual workflows, or configure new tools, or learn about project specific (unwritteN) conventions, are all things which stand in the way of succesful participation. The key value of the merge request workflow is the standardization across projects, such that it has immediate familiarity enabling people to contribution without having to read docs. This is a bit of an over simplication, since there is always some project specific knowledge required. Ideally this knowledge will be primarily around the code and not on the development process. I've increasingly recognised this in my own interactions with projects. Since the widespread adoption of Git, I find it a significant turn off to find a project is still using CVS, Subversion, or any other non-Git SCM. Increasingly I feel the same way about development workflow and the git forge. If I can fork a project on GitLab/GitHub, and open a pull request, I'm more likely to send a patch upstream, than if I have to register for a new account on a custom bug tracker or mailing list. On the flip side, while one of my own personal projects still has a mailing list, it is more enjoyable dealing with contributors if they open a pull request, as I see the CI results and know the patch can be applied with 1 click if desired. The more I use the tools, the more effective they become for me. Keeping new contributors interested ----------------------------------- The response to a new contributor is also important. The earlier someone engages with them, the better. With this in mind it is useful to be able to quickly identify new contributor and prioritize a response to them over other work. Automation (CI) can help in this respect by picking up & reporting problems without needing human attention. None the less, triage of incoming requests will be needed, primarily though tagging to classify incoming requests, and nominating certain people as reviewers. The power of the merge request model is that such triage is centralized & thus visible to participates, as opposed to being repeated by every individual when using email. People can see this triage and more effectively direct their attention where it is needed. The Web UI problem ------------------ The use of a web UI is often seen as the biggest downside to using a Git Forge, especially amongst people who are used to an email based code review workflow. This is a position I have great sympathy with, as I do largely prefer text based tools with strong keyboard navigation, over graphical tools with strong mouse navigation. The downsides of the web UI for merge requests becomes more apparent the larger the change is, especially if split across multiple patches. While common, this viewpoint on the merits of text UI over graphical UI is not universally held. The popularity of Git Forges amongst the open souce world demonstrates that many developers have no trouble with the web UI, or at least are willing to accept it in order to benefit from the other features offered. My preference for a text based UI has lead me to spend time investigating the API support for GitLab and from there developing a custom application for interacting with GitLab merge requests. https://gitlab.com/bichon-project/bichon The goal of the tool is to provide a user interface for merge requests that is familiar to users of the mutt email client. It is not especially advanced in its impl yet, and certainly has rough edges, but for me it has proved the general point that we can build useful tools to integrate with GitLab, which can mitigate the worst aspects of a Web UI. The need to build such a tool, however, has a cost as it diverts time away from working on libvirt itself, which negates some of the benefits of adopting GitLab in the first place. It is hoped, however, that this is a relatively short term cost that could ultimately be amoritized across multiple developers who use GitLab for their projects, while the benefits are repaid continuously over the long term. The review comment model ------------------------ Those familiar with a email based workflow using clients like mutt will often cherish the deeply threaded discussion model. It is worth bearing in mind that not all email clients will follow this, with many turning the threaded discussion into a flat discussion. The GitLab review comment system certainly doesn't offer the kind of threading that mutt does, but it is a step above what many email clients do. Against any single patch, it is possible to one or more standalone comments, and one or more discussion threads, attached to specific lines of code. The distinction between the two is whether replies are permitted. The threads are, however, flat, not deeply nested. There are two interesting and perhaps useful things to be aware of. First each discussion thread on a patch has a notion of whether it has been "resolved" or not. It is possible to (optionally) have threads auto-resolve when a new version of the patch series is posted to the merge request, or they can be resolved manually by a contributor. This can be valuable in keeping track of whether feedback on v1 was addressed in changes for v2, which is a challenge with email review workflows. The second interesting thing is that threads can be "rebased". This is both useful and annoying at the same time. What appears to happen behind the scenes is that the (commit hash, file, line num) that is associated with a comment thread can be auto-updated to point to the latest commit that contains the matching (file, line num). This is useful when a v2 is posted, as the comments from v1 are moved to v2, helping identify if the problem was addressed. It is annoying when dealing with patch series though, as it appears that a comment created on patch 3, might be reported against patch 15 when later queried, if patch 15 has context covering this same source file location. This makes some sense when considering the web UI, because by default it will show the combined diff of all patches in a series. It is less helpful for my bichon tool which is trying to present a per-patch view. In theory it should be possible to follow the diff context backwards to display the comment against every single patch which has context for the same line of code. This would in fact make it quite useful, but until that is done it is quite annoying. More generally, comments have nice integration with the rest of the toolset making it easy to refer to issue trackers, merge requests via magic markdown extensions, and to tag individual contributors to get their opinion on a question. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote: [...] Many of the things outlined below can be also read as "if you spend some effort in using the proper tools and using proper approaches" you can be more effective than the web-tools which are better than the default approach but far from great. Arguably you could apply same logic on the web based workflow too, but we are in a gravity well of having stuff set up to our liking. I also presume the 'merge request' workflow still supports doing fast-forward merges only as that would be a very big regression since we've kept linear history until now. At any rate, I hope you are right, and in the end it will be worth the hit on the workflow of existing contributors and reviewers who prefer email.

On Fri, Mar 06, 2020 at 01:35:48PM +0100, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
[...]
Many of the things outlined below can be also read as "if you spend some effort in using the proper tools and using proper approaches" you can be more effective than the web-tools which are better than the default approach but far from great. Arguably you could apply same logic on the web based workflow too, but we are in a gravity well of having stuff set up to our liking.
That is certainly a valid interpretation of some of the points. I think it applies more to long term contributors than to infrequent contributors. IOW, if you are a long term contributors you'll always benefit from investing time to optimize your work no matter what the tools. If you are an infrequent contributor the need to invest time to use the tools not a good tradeoff, especially if that's project-specific knowledge. Any of course that assumes it is even possible to use the tools in the first place, which we've seen is not the case for email, with contributors suffering lost mail.
I also presume the 'merge request' workflow still supports doing fast-forward merges only as that would be a very big regression since we've kept linear history until now.
Yes, both fast forward only, and merge commits are available as options. We'd obviously choose fast forward as our initial default. There is a third option, which is a hybrid of the two - merge commits with semi-linear history. This requires the code to be a fast-forward commit, but then creates a merge commit anyway. So you still effectively have linear history. The potential benefit of this is that the merge commit has a commit message that is machine generated that can include a link back to the merge request that triggered it - IOW an audit log of merges. This might make it appealing to use, but I've never tried it out, so can't say much more on pros/cons of it. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On 3/6/20 8:44 AM, Daniel P. Berrangé wrote: [...] What happens with this mailing list when the migration to the new workflow is completed with all the repos? Is it still going to be used for discussions, questions, RFCs and etcetera? I'd rather be in Gitlab watching opened issues and merge requests all the time, without the need to check the Libvirt ML ever again. And apparently we're leaning towards Gitlab. I'll not be standing here defending closed-source, Microsoft based Github, but I'm curious: aside from that (and that reason alone is enough, no need to grab the pitchforks), is there any other technical advantage for going Gitlab? I suppose most existing "coding support tools" are Github friendly already. Also, due to Microsoft deep pockets, Github will probably experience less downtime and have a better support overall in case something goes wrong. Thanks, DHB

On Fri, Mar 06, 2020 at 11:54:57AM -0300, Daniel Henrique Barboza wrote:
On 3/6/20 8:44 AM, Daniel P. Berrangé wrote:
[...]
What happens with this mailing list when the migration to the new workflow is completed with all the repos? Is it still going to be used for discussions, questions, RFCs and etcetera? I'd rather be in Gitlab watching opened issues and merge requests all the time, without the need to check the Libvirt ML ever again.
So we have a mixture of code submissions, design discussions (on libvir-list) and user discussions (on libvirt-users). The design discussions make up a fairly low volume of the mail. Based on what I see with other projects, they generally encourage design discussions to happen in the issue tracker. The designs may lead to many distinct pieces of work, each of which get turned into todo items and associated with separate merge requests. So the issue tracker is basically used to discuss / plan / coordinate all the non-coding work. With this approach there's not really much compelling reason to keep using the libvir-list. If you look at systemd, after they moved to merge requests, their mailing list still exists, but it mostly just has end user questions. For libvirt this role is filled by the libvirt-users mailing list. I wouldn't shut down libvir-list, but I wouldn't expect significant traffic in the long term.
And apparently we're leaning towards Gitlab. I'll not be standing here defending closed-source, Microsoft based Github, but I'm curious: aside from that (and that reason alone is enough, no need to grab the pitchforks), is there any other technical advantage for going Gitlab? I suppose most existing "coding support tools" are Github friendly already. Also, due to Microsoft deep pockets, Github will probably experience less downtime and have a better support overall in case something goes wrong.
The open source platform aspect is the overriding factor. I believe in the core value / message of open source software, so I can't credibly say that in order to develop open source software, you must use a closed source platform. It is a contradictory message IMHO. I would prefer if GitLab didn't follow the open-core model of split offerings, but it is at least still an open source project. At a technical level, I find the native CI integration more compelling in GitLab, vs GitHub where CI is traditionally outsourced to further 3rd party closed source services (though IIUC they have native Azure support now too as an option). This is fairly minor though. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

+1 on the initial thread b
On 6 Mar 2020, at 15:54, Daniel Henrique Barboza <danielhb413@gmail.com> wrote:
On 3/6/20 8:44 AM, Daniel P. Berrangé wrote:
[...]
What happens with this mailing list when the migration to the new workflow is completed with all the repos? Is it still going to be used for discussions, questions, RFCs and etcetera? I'd rather be in Gitlab watching opened issues and merge requests all the time, without the need to check the Libvirt ML ever again.
And apparently we're leaning towards Gitlab. I'll not be standing here defending closed-source, Microsoft based Github, but I'm curious: aside from that (and that reason alone is enough, no need to grab the pitchforks), is there any other technical advantage for going Gitlab? I suppose most existing "coding support tools" are Github friendly already. Also, due to Microsoft deep pockets, Github will probably experience less downtime and have a better support overall in case something goes wrong.
GitHub and GitLab have different approaches to CI, with pros and cons on both sides. Obviously, it is easier to get stuff tested on Windows with GitHub, for example. You can use both, with automatic mirroring of the commits. For some of my own projects, I have dual push targets in git (triple, actually, SourceForge), and then I get two sets of (different) CI tests on a push. For example, GitLab will test a number of Linux targets like Ubuntu, etc, while GitHub will test macOS and someday Windows. I am not aware of a good way to sync issues, though, only commits. Anybody knows differently?
Thanks,
DHB

On Tue, Mar 17, 2020 at 12:06:45PM +0100, Christophe de Dinechin wrote:
+1 on the initial thread b
On 6 Mar 2020, at 15:54, Daniel Henrique Barboza <danielhb413@gmail.com> wrote:
On 3/6/20 8:44 AM, Daniel P. Berrangé wrote:
[...]
What happens with this mailing list when the migration to the new workflow is completed with all the repos? Is it still going to be used for discussions, questions, RFCs and etcetera? I'd rather be in Gitlab watching opened issues and merge requests all the time, without the need to check the Libvirt ML ever again.
And apparently we're leaning towards Gitlab. I'll not be standing here defending closed-source, Microsoft based Github, but I'm curious: aside from that (and that reason alone is enough, no need to grab the pitchforks), is there any other technical advantage for going Gitlab? I suppose most existing "coding support tools" are Github friendly already. Also, due to Microsoft deep pockets, Github will probably experience less downtime and have a better support overall in case something goes wrong.
GitHub and GitLab have different approaches to CI, with pros and cons on both sides. Obviously, it is easier to get stuff tested on Windows with GitHub, for example.
You can use both, with automatic mirroring of the commits. For some of my own projects, I have dual push targets in git (triple, actually, SourceForge), and then I get two sets of (different) CI tests on a push. For example, GitLab will test a number of Linux targets like Ubuntu, etc, while GitHub will test macOS and someday Windows.
NB, windows testing is covered by mingw64 cross compilers, at least for build testing. If we wanted to run unit tests we can use wine, but it hasn't been a priority. For real functional tests we'd need a real windows install, but that's even further down the list of things we care about. macOS testing we get via Travis and that's the main gap we have with GitLab, unless we get access some hardware we can setup as a GitLab custom CI runner.
I am not aware of a good way to sync issues, though, only commits. Anybody knows differently?
Syncing issues & merge requests is tricky to do accurately, because you need to parse the comments to identify references to usernames and issues, etc. IMHO it just isn't worth the hassle to. Syncing everything would also make google search results and split attention with people not being sure which is the master vs mirror. We already have that problem to some extent with the existing commit mirroring, so I'm loathe to make it more confusing. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so, but never made a concrete decision.
One other thing that worries me about this is that we've finally established a way close to qemu developers for notifying us if they are going to deprecate something or change something important. With moving development to some random web page with non-standard interfaces this will just mean that the notifications in this process will either stay on the old mailing list or be forgotten if we don't act on them. Moving development to some other place will in this regard just mean that we'll have to watch two places at the same time. While this seems to be a very low impact thing, the advantages of the new process you've outlined will only ever apply to drive-by contributors. Anybody wanting to take it seriously will necessarily need to subscribe to the mailing list anyways. In the end I just don't want to destroy the relationship with qemu developers by not acting on the notifications of change they send to us.

On 17. 3. 2020 8:52, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so, but never made a concrete decision.
One other thing that worries me about this is that we've finally established a way close to qemu developers for notifying us if they are going to deprecate something or change something important.
With moving development to some random web page with non-standard interfaces this will just mean that the notifications in this process will either stay on the old mailing list or be forgotten if we don't act on them.
Moving development to some other place will in this regard just mean that we'll have to watch two places at the same time.
While this seems to be a very low impact thing, the advantages of the new process you've outlined will only ever apply to drive-by contributors. Anybody wanting to take it seriously will necessarily need to subscribe to the mailing list anyways.
In the end I just don't want to destroy the relationship with qemu developers by not acting on the notifications of change they send to us.
I don't think I share this view. The way qemu developers notify us is cross-posting to libvir-list. They can still do that and with the traffic on the list going down it will be pretty easy to spot these cross posts. Or am I missing something? Michal

On Tue, Mar 17, 2020 at 09:20:07 +0100, Michal Privoznik wrote:
On 17. 3. 2020 8:52, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so, but never made a concrete decision.
One other thing that worries me about this is that we've finally established a way close to qemu developers for notifying us if they are going to deprecate something or change something important.
With moving development to some random web page with non-standard interfaces this will just mean that the notifications in this process will either stay on the old mailing list or be forgotten if we don't act on them.
Moving development to some other place will in this regard just mean that we'll have to watch two places at the same time.
While this seems to be a very low impact thing, the advantages of the new process you've outlined will only ever apply to drive-by contributors. Anybody wanting to take it seriously will necessarily need to subscribe to the mailing list anyways.
In the end I just don't want to destroy the relationship with qemu developers by not acting on the notifications of change they send to us.
I don't think I share this view. The way qemu developers notify us is cross-posting to libvir-list. They can still do that and with the traffic on the list going down it will be pretty easy to spot these cross posts. Or am I missing something?
Yes. As mentioned above though you need to be subscribed to the list though. Also as mentioned above, that means that any serious developer will need to be subscribe to the list. So all the point of not having to subscribe to the list applies only to drive-by contributors.

On 17. 3. 2020 9:26, Peter Krempa wrote:
On Tue, Mar 17, 2020 at 09:20:07 +0100, Michal Privoznik wrote:
On 17. 3. 2020 8:52, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so, but never made a concrete decision.
One other thing that worries me about this is that we've finally established a way close to qemu developers for notifying us if they are going to deprecate something or change something important.
With moving development to some random web page with non-standard interfaces this will just mean that the notifications in this process will either stay on the old mailing list or be forgotten if we don't act on them.
Moving development to some other place will in this regard just mean that we'll have to watch two places at the same time.
While this seems to be a very low impact thing, the advantages of the new process you've outlined will only ever apply to drive-by contributors. Anybody wanting to take it seriously will necessarily need to subscribe to the mailing list anyways.
In the end I just don't want to destroy the relationship with qemu developers by not acting on the notifications of change they send to us.
I don't think I share this view. The way qemu developers notify us is cross-posting to libvir-list. They can still do that and with the traffic on the list going down it will be pretty easy to spot these cross posts. Or am I missing something?
Yes. As mentioned above though you need to be subscribed to the list though. Also as mentioned above, that means that any serious developer will need to be subscribe to the list. So all the point of not having to subscribe to the list applies only to drive-by contributors.
Sure, but I thought this is expected. Every project has a place to discuss ideas, make decisions. Some use pull requests to do that (please don't), some have a mailing list. I see libvirt in the latter group. And to some extent, we are already in that situation. I mean, if I were a drive by contributor, I would subscribe to the list, post my patches and ignore the rest of incoming e-mails from the list. Maybe I'm misunderstanding and what is suggested it to move even all the discussion to gitlab. If that is the case, I stand by you. We should not do that. But if we are moving just the repo then I guess it is fine. Michal

On Tue, Mar 17, 2020 at 09:42:30 +0100, Michal Privoznik wrote:
On 17. 3. 2020 9:26, Peter Krempa wrote:
On Tue, Mar 17, 2020 at 09:20:07 +0100, Michal Privoznik wrote:
On 17. 3. 2020 8:52, Peter Krempa wrote:
[...]
I don't think I share this view. The way qemu developers notify us is cross-posting to libvir-list. They can still do that and with the traffic on the list going down it will be pretty easy to spot these cross posts. Or am I missing something?
Yes. As mentioned above though you need to be subscribed to the list though. Also as mentioned above, that means that any serious developer will need to be subscribe to the list. So all the point of not having to subscribe to the list applies only to drive-by contributors.
Sure, but I thought this is expected. Every project has a place to discuss ideas, make decisions. Some use pull requests to do that (please don't), some have a mailing list. I see libvirt in the latter group. And to some extent, we are already in that situation. I mean, if I were a drive by contributor, I would subscribe to the list, post my patches and ignore the rest of incoming e-mails from the list.
Maybe I'm misunderstanding and what is suggested it to move even all the discussion to gitlab. If that is the case, I stand by you. We should not do that. But if we are moving just the repo then I guess it is fine.
See the second paragraph of another subthread of this discussion: https://www.redhat.com/archives/libvir-list/2020-March/msg00196.html I wanted to raise this issue separately so that it's not burried. At any rate, the idea of switching to the forge is to stop using email. While this might appeal to upper layer developers where the cool new deveopment approaches are used, we should not jeopardize our relationship with qemu as we are trying to offer them our value especially in the region of deprecation and changes of interfaces which we are making transparent to users. And if we are not going to kill off the libvir-list, the wins described in the original thread are not as clear cut, as it just adds things you have to watch/deal with, but for any real work, you'll have to endure the pain from both email and web-based forges.

On Tue, Mar 17, 2020 at 10:01:24AM +0100, Peter Krempa wrote:
On Tue, Mar 17, 2020 at 09:42:30 +0100, Michal Privoznik wrote:
On 17. 3. 2020 9:26, Peter Krempa wrote:
On Tue, Mar 17, 2020 at 09:20:07 +0100, Michal Privoznik wrote:
On 17. 3. 2020 8:52, Peter Krempa wrote:
[...]
I don't think I share this view. The way qemu developers notify us is cross-posting to libvir-list. They can still do that and with the traffic on the list going down it will be pretty easy to spot these cross posts. Or am I missing something?
Yes. As mentioned above though you need to be subscribed to the list though. Also as mentioned above, that means that any serious developer will need to be subscribe to the list. So all the point of not having to subscribe to the list applies only to drive-by contributors.
Sure, but I thought this is expected. Every project has a place to discuss ideas, make decisions. Some use pull requests to do that (please don't), some have a mailing list. I see libvirt in the latter group. And to some extent, we are already in that situation. I mean, if I were a drive by contributor, I would subscribe to the list, post my patches and ignore the rest of incoming e-mails from the list.
Maybe I'm misunderstanding and what is suggested it to move even all the discussion to gitlab. If that is the case, I stand by you. We should not do that. But if we are moving just the repo then I guess it is fine.
See the second paragraph of another subthread of this discussion:
https://www.redhat.com/archives/libvir-list/2020-March/msg00196.html
I wanted to raise this issue separately so that it's not burried. At any rate, the idea of switching to the forge is to stop using email.
While this might appeal to upper layer developers where the cool new deveopment approaches are used, we should not jeopardize our relationship with qemu as we are trying to offer them our value especially in the region of deprecation and changes of interfaces which we are making transparent to users.
And if we are not going to kill off the libvir-list, the wins described in the original thread are not as clear cut, as it just adds things you have to watch/deal with, but for any real work, you'll have to endure the pain from both email and web-based forges.
Let me quote the paragraph in question:
Based on what I see with other projects, they generally encourage design discussions to happen in the issue tracker. The designs may lead to many distinct pieces of work, each of which get turned into todo items and associated with separate merge requests. So the issue tracker is basically used to discuss / plan / coordinate all the non-coding work. With this approach there's not really much compelling reason to keep using the libvir-list. If you look at systemd, after they moved to merge requests, their mailing list still exists, but it mostly just has end user questions. For libvirt this role is filled by the libvirt-users mailing list. I wouldn't shut down libvir-list, but I wouldn't expect significant traffic in the long term.
If I understand it correctly there is no plan to kill libvir-list. I completely support your concern about collaboration with QEMU developers but I don't thing this move will affect that in any way. Yes, existing core developers would probably have to watch two places for libvirt development and yes new core developers would have to subscribe to libvir-list as well if they need to cooperate with QEMU, but I don't thing the real impact on existing core developers is that significant. If we move to GitLab I guess that most of us will configure email based notifications for new pull requests or issues and since it's an email you can do whatever you like with it, for example filter it into the same folder as libvir-list. Pavel

On Tue, Mar 17, 2020 at 09:26:44AM +0100, Peter Krempa wrote:
On Tue, Mar 17, 2020 at 09:20:07 +0100, Michal Privoznik wrote:
On 17. 3. 2020 8:52, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so, but never made a concrete decision.
One other thing that worries me about this is that we've finally established a way close to qemu developers for notifying us if they are going to deprecate something or change something important.
With moving development to some random web page with non-standard interfaces this will just mean that the notifications in this process will either stay on the old mailing list or be forgotten if we don't act on them.
Moving development to some other place will in this regard just mean that we'll have to watch two places at the same time.
While this seems to be a very low impact thing, the advantages of the new process you've outlined will only ever apply to drive-by contributors. Anybody wanting to take it seriously will necessarily need to subscribe to the mailing list anyways.
In the end I just don't want to destroy the relationship with qemu developers by not acting on the notifications of change they send to us.
I don't think I share this view. The way qemu developers notify us is cross-posting to libvir-list. They can still do that and with the traffic on the list going down it will be pretty easy to spot these cross posts. Or am I missing something?
Yes. As mentioned above though you need to be subscribed to the list though. Also as mentioned above, that means that any serious developer will need to be subscribe to the list. So all the point of not having to subscribe to the list applies only to drive-by contributors.
Our long term contributors are the only ones who are likely to take any actions based on the QEMU cross-posted messages, so I don't think we'll loose anything measurable in this regard. We could probably do with formalizing our handling of QEMU deprecations too. There have been a couple of occassions where we saw the message but then failed to take action. It might be worth us explicitly filing an issue against libvirt for every deprecation warning, so that we know what is outtstanding on our todo list in this regard. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Tue, Mar 17, 2020 at 09:47:22 +0000, Daniel Berrange wrote:
On Tue, Mar 17, 2020 at 09:26:44AM +0100, Peter Krempa wrote:
On Tue, Mar 17, 2020 at 09:20:07 +0100, Michal Privoznik wrote:
On 17. 3. 2020 8:52, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so, but never made a concrete decision.
One other thing that worries me about this is that we've finally established a way close to qemu developers for notifying us if they are going to deprecate something or change something important.
With moving development to some random web page with non-standard interfaces this will just mean that the notifications in this process will either stay on the old mailing list or be forgotten if we don't act on them.
Moving development to some other place will in this regard just mean that we'll have to watch two places at the same time.
While this seems to be a very low impact thing, the advantages of the new process you've outlined will only ever apply to drive-by contributors. Anybody wanting to take it seriously will necessarily need to subscribe to the mailing list anyways.
In the end I just don't want to destroy the relationship with qemu developers by not acting on the notifications of change they send to us.
I don't think I share this view. The way qemu developers notify us is cross-posting to libvir-list. They can still do that and with the traffic on the list going down it will be pretty easy to spot these cross posts. Or am I missing something?
Yes. As mentioned above though you need to be subscribed to the list though. Also as mentioned above, that means that any serious developer will need to be subscribe to the list. So all the point of not having to subscribe to the list applies only to drive-by contributors.
Our long term contributors are the only ones who are likely to take any actions based on the QEMU cross-posted messages, so I don't think we'll loose anything measurable in this regard.
Sounds a bit hypocritical then. The merge request workflow is inconveniencing long term contributors so that we can attract new contributors. If the new contributors aren't seeing those they won't be able to act on them. And if we don't expect new contributors to be involved to that extent, why are we even doing this?
We could probably do with formalizing our handling of QEMU deprecations too. There have been a couple of occassions where we saw the message buta
I think that requiring qemu developers to sign up to some web-page to file something are going to be detremental to the effort. The notifications are based mostly on the fact that changes to the file documenting the deprecations are automatically CCd to libvirt-list. This way if a drive-by qemu contributor sends patches they might not care enough to go through the notification process in the first place.
then failed to take action. It might be worth us explicitly filing an issue against libvirt for every deprecation warning, so that we know what is outtstanding on our todo list in this regard.
There are also instances, where we've seen the message, filed bugs (issues) and not taken action. It's always human factor. There are at least these that I know of and there wasn't any action taken: https://bugzilla.redhat.com/show_bug.cgi?id=1783355 https://bugzilla.redhat.com/show_bug.cgi?id=1717899 I didn't even bother looking into the upstream bugzilla btw. There's a giant pile of stuff which was filed but nobody cares about. I'm not sure if we want to mirror that to whatever-hub/lab but either way given some time I feel it will end up the same way as the upstream bugzilla.

On Tue, Mar 17, 2020 at 11:12:54AM +0100, Peter Krempa wrote:
On Tue, Mar 17, 2020 at 09:47:22 +0000, Daniel Berrange wrote:
On Tue, Mar 17, 2020 at 09:26:44AM +0100, Peter Krempa wrote:
On Tue, Mar 17, 2020 at 09:20:07 +0100, Michal Privoznik wrote:
On 17. 3. 2020 8:52, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so, but never made a concrete decision.
One other thing that worries me about this is that we've finally established a way close to qemu developers for notifying us if they are going to deprecate something or change something important.
With moving development to some random web page with non-standard interfaces this will just mean that the notifications in this process will either stay on the old mailing list or be forgotten if we don't act on them.
Moving development to some other place will in this regard just mean that we'll have to watch two places at the same time.
While this seems to be a very low impact thing, the advantages of the new process you've outlined will only ever apply to drive-by contributors. Anybody wanting to take it seriously will necessarily need to subscribe to the mailing list anyways.
In the end I just don't want to destroy the relationship with qemu developers by not acting on the notifications of change they send to us.
I don't think I share this view. The way qemu developers notify us is cross-posting to libvir-list. They can still do that and with the traffic on the list going down it will be pretty easy to spot these cross posts. Or am I missing something?
Yes. As mentioned above though you need to be subscribed to the list though. Also as mentioned above, that means that any serious developer will need to be subscribe to the list. So all the point of not having to subscribe to the list applies only to drive-by contributors.
Our long term contributors are the only ones who are likely to take any actions based on the QEMU cross-posted messages, so I don't think we'll loose anything measurable in this regard.
Sounds a bit hypocritical then. The merge request workflow is inconveniencing long term contributors so that we can attract new contributors. If the new contributors aren't seeing those they won't be able to act on them. And if we don't expect new contributors to be involved to that extent, why are we even doing this?
It isn't hypocritical, it is pragmatic approach to the interaction we have. By turning deprecations into issues they become visible to all developers.
We could probably do with formalizing our handling of QEMU deprecations too. There have been a couple of occassions where we saw the message buta
I think that requiring qemu developers to sign up to some web-page to file something are going to be detremental to the effort. The notifications are based mostly on the fact that changes to the file documenting the deprecations are automatically CCd to libvirt-list.
I'm not suggesting QEMU devs file the issues, I'm saying that when libvirt is copied on a deprecation alert, we should file a issue against libvirt ourselves to track it (assuming it affects libvirt)
then failed to take action. It might be worth us explicitly filing an issue against libvirt for every deprecation warning, so that we know what is outtstanding on our todo list in this regard.
There are also instances, where we've seen the message, filed bugs (issues) and not taken action. It's always human factor.
There are at least these that I know of and there wasn't any action taken:
https://bugzilla.redhat.com/show_bug.cgi?id=1783355 https://bugzilla.redhat.com/show_bug.cgi?id=1717899
I didn't even bother looking into the upstream bugzilla btw. There's a giant pile of stuff which was filed but nobody cares about. I'm not sure if we want to mirror that to whatever-hub/lab but either way given some time I feel it will end up the same way as the upstream bugzilla.
We need todo better at bug triage in general - ignoring stuff just makes the problem worse. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so,
One thing I feel the need to voice until this is taken in place is a matter of personal preference: I severely dislike the merge request workflow and I'll be severely disappointed once we switch over to it.

On 3/26/20 10:10 AM, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so,
One thing I feel the need to voice until this is taken in place is a matter of personal preference:
I severely dislike the merge request workflow and I'll be severely disappointed once we switch over to it.
I'm not thrilled about it either, but I believe that once we get used to the new workflow we'll learn to appreciate all the possibilities a GitForge tool brings to the table. One point that I already made earlier in the thread: having to track discussions in two places (mailing list and Gitlab) will be annoying. I understand that the idea is that we'll organically move move all possible discussions to Gitlab. I'd rather make it a blunt and quick directive: once we moved to Gitlab, mailing list is completely discouraged unless it's a topic that needs to be discussed together with the QEMU or any other mailing list. If we leave to personal preference we'll have people posting stuff everywhere. DHB

On Thu, Mar 26, 2020 at 11:22:11 -0300, Daniel Henrique Barboza wrote:
On 3/26/20 10:10 AM, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so,
One thing I feel the need to voice until this is taken in place is a matter of personal preference:
I severely dislike the merge request workflow and I'll be severely disappointed once we switch over to it.
Thank you for your opinion!
I'm not thrilled about it either, but I believe that once we get used to the new workflow we'll learn to appreciate all the possibilities a GitForge tool brings to the table.
IMO the price to pay for the few good features is too high, but again that's very subjective in my opinion as I can't counter with credible technical points against it. That says that the problems outlined are real, but the replacement is in no way a clear win either. It solves some of the pain points of the current process, but it makes the very few, but important things such as review way more painful. The unfortunate part is, that I don't feel that the tradeoff is worth it. And yes, that's an egoistic belief as I worked around the quirks of the existing process already.
One point that I already made earlier in the thread: having to track discussions in two places (mailing list and Gitlab) will be annoying. I understand that the idea is that we'll organically move move all possible discussions to Gitlab. I'd rather make it a blunt and quick directive: once we moved to Gitlab, mailing list is completely discouraged unless it's a topic that needs to be discussed together with the QEMU or any other mailing list. If we leave to personal preference we'll have people posting stuff everywhere.
I agree that there should be one place only for discussions. I feel though that using gitlab for discussions will also be a severe regression in usability. Again this is from a grumpy guy who likes the old-school.

On Thu, Mar 26, 2020 at 11:22:11AM -0300, Daniel Henrique Barboza wrote:
On 3/26/20 10:10 AM, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so,
One thing I feel the need to voice until this is taken in place is a matter of personal preference:
I severely dislike the merge request workflow and I'll be severely disappointed once we switch over to it.
I'm not thrilled about it either, but I believe that once we get used to the new workflow we'll learn to appreciate all the possibilities a GitForge tool brings to the table.
FYI, every time you post to this list, there are a bunch of subscribers whose email system (bogusly) rejects your messages, so as list admin I get a bunch of bounces like: [quote] ==================================================== Your message couldn't be delivered ==================================================== The message you sent to [REDACTED] couldn't be delivered due to: Recipient email server rejected the message. ––––––––––––––––––––––––––––––––––––––––––––––––––– Further Information Rejected by header based Blocked Senders: danielhb413@gmail.com [/quote] AFAIK, there's nothing wrong with what you're sending - this is a bad mail server that has inexplicably put your email address on a delivery blacklist :-( The result though is that a bunch of list subscribers never see any email or patches that you send, and worse, they probably don't even realize that they're missing them. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On 3/26/20 12:47 PM, Daniel P. Berrangé wrote:
On Thu, Mar 26, 2020 at 11:22:11AM -0300, Daniel Henrique Barboza wrote:
On 3/26/20 10:10 AM, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
[...]
FYI, every time you post to this list, there are a bunch of subscribers whose email system (bogusly) rejects your messages, so as list admin I get a bunch of bounces like:
[quote] ====================================================
Your message couldn't be delivered
====================================================
The message you sent to [REDACTED] couldn't be delivered due to: Recipient email server rejected the message.
––––––––––––––––––––––––––––––––––––––––––––––––––– Further Information
Rejected by header based Blocked Senders: danielhb413@gmail.com [/quote]
AFAIK, there's nothing wrong with what you're sending - this is a bad mail server that has inexplicably put your email address on a delivery blacklist :-(
The result though is that a bunch of list subscribers never see any email or patches that you send, and worse, they probably don't even realize that they're missing them.
Ironically I'm using this gmail because IBM corporate email has bouncing problems with community mailing lists since forever, and the IMAP email IBM provides uses a similiar structure Red Hat uses (MX gateway and etc) and it has .... problems. And now you're saying that my gmail is being flagged as spam. That's nice. So, in the end, I should be one advocating for a stronger Gitlab push. 2020 gets weirder and weirder. DHB
Regards, Daniel

On Thu, Mar 26, 2020 at 02:10:59PM +0100, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so,
One thing I feel the need to voice until this is taken in place is a matter of personal preference:
I severely dislike the merge request workflow and I'll be severely disappointed once we switch over to it.
Can you elaborate on specific things you don't like, so we can see if there are any options to mitigate them ? Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On Thu, Mar 26, 2020 at 15:50:20 +0000, Daniel Berrange wrote:
On Thu, Mar 26, 2020 at 02:10:59PM +0100, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so,
One thing I feel the need to voice until this is taken in place is a matter of personal preference:
I severely dislike the merge request workflow and I'll be severely disappointed once we switch over to it.
Can you elaborate on specific things you don't like, so we can see if there are any options to mitigate them ?
- it's a web page - it's very slow on big projects and slow in general - it's ugly, "theme" support is a joke, white background burns my eyes - it works badly on a portrait display - mangles commit messages in an attempt to render them (link and signed-off line concatenated: https://gitlab.com/libvirt/libvirt/-/commit/e05dd1abdc3b3eeac6e12ab105e56138... - usability - default view after opening a branch is "Files" not "commits" - web interface doesn't really carry information about which merge requests are new to me - review is terrible - threads are only single level - response can't be quoted (okay, they can but you have to copy and pase what you want to quote and then select that it's a quote) - you need to click to see the code - comments to code are in proportional font - a lot of useless clutter in the UI - need to click open comments in code - extra steps necessary to apply code locally (granted you get a branch by default, but fetching it is not as easy as I have with mail workflow) - review process favours review without actually fetching the code locally (if you can click a button, who will actually test that the code works?) Now I know you will suggest I try your 'bichon' tool for reviews as it fixes a handful of the problems outlined above, but unfortunately that comes with it's own set of problems since it's in infancy stage. In addition with current upstream bug tracker and discussions at least being delivered using email I have notifications about new stuff even if I have to interact with a web page. Bichon doesn't seem to want to deal with 'issues' section at all. That means that I'll have to trade a process that works very well for me for using either a web page with many problems of it's own or aleviate some of the problems to command line tool which is being developed rapidly and thus likely to break often. And some of the problems are not addressed by either, e.g. quoting parts of comments. All together this will be a substantial downgrade for me and thus I'm unhappy.

On Fri, Mar 27, 2020 at 02:15:01PM +0100, Peter Krempa wrote:
On Thu, Mar 26, 2020 at 15:50:20 +0000, Daniel Berrange wrote:
On Thu, Mar 26, 2020 at 02:10:59PM +0100, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so,
One thing I feel the need to voice until this is taken in place is a matter of personal preference:
I severely dislike the merge request workflow and I'll be severely disappointed once we switch over to it.
Can you elaborate on specific things you don't like, so we can see if there are any options to mitigate them ?
- it's a web page - it's very slow on big projects and slow in general - it's ugly, "theme" support is a joke, white background burns my eyes - it works badly on a portrait display - mangles commit messages in an attempt to render them (link and signed-off line concatenated: https://gitlab.com/libvirt/libvirt/-/commit/e05dd1abdc3b3eeac6e12ab105e56138...
This looks like a bug in their code which renders the messages. It is failing to preserve a newline character when it follows a hyperlink. I'll see about reporting this as an issue.
- usability - default view after opening a branch is "Files" not "commits" - web interface doesn't really carry information about which merge requests are new to me
- review is terrible - threads are only single level - response can't be quoted (okay, they can but you have to copy and pase what you want to quote and then select that it's a quote) - you need to click to see the code - comments to code are in proportional font - a lot of useless clutter in the UI - need to click open comments in code - extra steps necessary to apply code locally (granted you get a branch by default, but fetching it is not as easy as I have with mail workflow)
Overall the issues are primarily focused on the web UI, rather than the conceptual idea of the merge request workflow. I share the same view of many of these issues, but I do think we can mitigate most of it with a terminal based tool as an option for those who don't like the web UI.
- review process favours review without actually fetching the code locally (if you can click a button, who will actually test that the code works?)
If anything I feel the current email review makes me even less likely to actually apply & test the code locally, as it requires several tedious error prone steps, compared to being able to direct "git fetch" a ref for the merge request. The more automated testing we do the better, but either way we rely on reviewers to be diligent enough to decide when something needs some more testing.
Now I know you will suggest I try your 'bichon' tool for reviews as it fixes a handful of the problems outlined above, but unfortunately that comes with it's own set of problems since it's in infancy stage.
Yes, it certainly isn't developed to where it needs to be yet, but I have started using it for real code review on libosinfo repos and found it is useful already.
In addition with current upstream bug tracker and discussions at least being delivered using email I have notifications about new stuff even if I have to interact with a web page. Bichon doesn't seem to want to deal with 'issues' section at all.
GitLab issues can be delivered via email just like Bugzilla, though IIRC you'll need to opt-in for any repositories you care about (if you're not the repo owner). In the main page for a repo https://gitlab.com/libvirt/libvirt/ Next to the repo title there is a bell icon where you can control what level of email notifications you get. Your profile page: https://gitlab.com/profile/notifications will show what repos you're getting notifications on currently Fabiano did file an RFE suggesting that we should support the issue tracker in Bichon too. I think that is a sensible idea, but personally I'm going to focus on merge requests right now so that's where the biggest wins will be. In terms of Web UI for issues, it is already better than Bugzilla as it is way simpler without all the RHEL tracking junk. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

On a Friday in 2020, Daniel P. Berrangé wrote:
On Fri, Mar 27, 2020 at 02:15:01PM +0100, Peter Krempa wrote:
On Thu, Mar 26, 2020 at 15:50:20 +0000, Daniel Berrange wrote:
On Thu, Mar 26, 2020 at 02:10:59PM +0100, Peter Krempa wrote:
On Fri, Mar 06, 2020 at 11:44:07 +0000, Daniel Berrange wrote:
We've discussed the idea of replacing our mailing list review workflow with a merge request workflow in various places, over the last 6 months or so,
One thing I feel the need to voice until this is taken in place is a matter of personal preference:
I severely dislike the merge request workflow and I'll be severely disappointed once we switch over to it.
Can you elaborate on specific things you don't like, so we can see if there are any options to mitigate them ?
- it's a web page - it's very slow on big projects and slow in general - it's ugly, "theme" support is a joke, white background burns my eyes - it works badly on a portrait display - mangles commit messages in an attempt to render them (link and signed-off line concatenated: https://gitlab.com/libvirt/libvirt/-/commit/e05dd1abdc3b3eeac6e12ab105e56138...
This looks like a bug in their code which renders the messages. It is failing to preserve a newline character when it follows a hyperlink. I'll see about reporting this as an issue.
- usability - default view after opening a branch is "Files" not "commits" - web interface doesn't really carry information about which merge requests are new to me
- review is terrible - threads are only single level - response can't be quoted (okay, they can but you have to copy and pase what you want to quote and then select that it's a quote) - you need to click to see the code - comments to code are in proportional font - a lot of useless clutter in the UI - need to click open comments in code - extra steps necessary to apply code locally (granted you get a branch by default, but fetching it is not as easy as I have with mail workflow)
Overall the issues are primarily focused on the web UI, rather than the conceptual idea of the merge request workflow.
Well, many of the points in the original mail were actually against mailing lists and not the simple merge-request-free workflow we are using thanks to them :)
I share the same view of many of these issues, but I do think we can mitigate most of it with a terminal based tool as an option for those who don't like the web UI.
- review process favours review without actually fetching the code locally (if you can click a button, who will actually test that the code works?)
If anything I feel the current email review makes me even less likely to actually apply & test the code locally, as it requires several tedious error prone steps, compared to being able to direct "git fetch" a ref for the merge request.
The more automated testing we do the better, but either way we rely on reviewers to be diligent enough to decide when something needs some more testing.
Now I know you will suggest I try your 'bichon' tool for reviews as it fixes a handful of the problems outlined above, but unfortunately that comes with it's own set of problems since it's in infancy stage.
Yes, it certainly isn't developed to where it needs to be yet, but I have started using it for real code review on libosinfo repos and found it is useful already.
Oh it already does stuff? Time to file some issues then because I could not get it to do anything useful. Jano

On Fri, 2020-03-27 at 14:15 +0100, Peter Krempa wrote:
- review is terrible - threads are only single level - response can't be quoted (okay, they can but you have to copy and pase what you want to quote and then select that it's a quote)
FYI, you can just select the part you're interested in and then hit 'r'. -- Andrea Bolognani / Red Hat / Virtualization
participants (8)
-
Andrea Bolognani
-
Christophe de Dinechin
-
Daniel Henrique Barboza
-
Daniel P. Berrangé
-
Ján Tomko
-
Michal Prívozník
-
Pavel Hrdina
-
Peter Krempa