[libvirt] Publishing libvirt 'patches' JSON database?
I'd imagine most developers on this list have their own workflows to track/apply/test patches from the mailing list. I normally just save patches in Mutt's Maildir and apply them manually to a Git branch. I recently began using the 'patches'[1] tool, that some of you on this list must be familiar with, to test arbitrary QEMU patches from the mailing list, and found it relatively easy large patch series from mailing list. I'm writing this to check if the libvirt upstream community finds it desirable to setup such a patches database. What is 'patches'? ------------------ Quoting the README[2]: "patches is a patch tracking system. It consists of two parts: a set of commands that build a database of patches from a mailing list and then a set of commands that can search that database. It supports the following features: - Tracking patch status by determining which patches are already committed, have newer versions posted, are RFC, etc. - Applying patches or pull requests with a single command. - Searching for patches using a rich query language." A quick workflow to test patches from mailing list --------------------------------------------------- FWIW, this is my rudimentary use-case. Clone the repo, and fetch the patches database (may take a few minutes): $ git clone https://github.com/stefanha/patches.git $ cd patches $ ./patches fetch http://wiki.qemu.org/patches/patches.json And, suppose, I want to test Dan's LUKS encryption support (26) patch series, I go find the message ID: $ ./patches list | grep -B5 "Support LUKS encryption" Message-id: 1456747261-22032-1-git-send-email-berrange@redhat.com From: Daniel P. Berrange <berrange@redhat.com> Date: 2016-02-29 Tags: v4 [0/26] Support LUKS encryption in block devices Trivially apply it (assuming you have a test branch): $ ./patches apply --git-dir ~/src/qemu/ \ id:1456747261-22032-1-git-send-email-berrange@redhat.com Applying: crypto: add cryptographic random byte source Applying: crypto: add support for PBKDF2 algorithm Applying: crypto: add support for generating initialization vectors Applying: crypto: add support for anti-forensic split algorithm Applying: crypto: skip testing of unsupported cipher algorithms [...] Test away! Rinse, repeat: "The `fetch` command should be run whenever you want to refresh the patch database; it will download any new patches that have arrived since the last time you did a fetch." Here's[3] the documentation to setup the patches database. [1] https://github.com/stefanha/patches/tree/stefanha-tweaks [2] https://github.com/stefanha/patches/blob/stefanha-tweaks/README.md [3] https://github.com/stefanha/patches/blob/stefanha-tweaks/README-server.md -- /kashyap
On 03/23/2016 11:34 AM, Kashyap Chamarthy wrote:
I'd imagine most developers on this list have their own workflows to track/apply/test patches from the mailing list. I normally just save patches in Mutt's Maildir and apply them manually to a Git branch. I recently began using the 'patches'[1] tool, that some of you on this list must be familiar with, to test arbitrary QEMU patches from the mailing list, and found it relatively easy large patch series from mailing list.
I'm writing this to check if the libvirt upstream community finds it desirable to setup such a patches database.
I haven't played much with 'patches' so I can't give a first hand opinion. But if it's low effort to set up and maintain seems like a no-brainer to generate and publish the metadata so libvirt devs can start playing with it. Although your details don't cover it, one of the major benefits of 'patches' is that it also scoops up metadata like Reviewed-by:, and gives an easy way to get those tags into commit messages. Good for giving prolific reviewers credit (and shaming those that don't do enough reviews :) ). The corollary is that you can search for unreviewed series, which IMO is desperately needed for libvir-list; the review time is pretty bad and I can't begin to count the number of drive-by patches we've received on list over the years that go completely unresponded. And I say this knowing full well I probably have the worst review-to-patches-authored ratio of all libvirt committers... but I don't live on libvir-list and often go 3-6 months at a time where I don't have any libvirt patches, and usually during those times I just skim-and-archive the list contents. If we had a programmatic tool to show me the unreviewed patches, I could run that on occasion and pitch in much more easily, at least reviewing bits that don't need a lot of context. We could even have an informal policy like 'theres XXX number of unreviewed patches on the list... no @redhat people post patches until we review the backlog' What if a patch series is NACKd or abandoned, is there a way to get it out of the patches metadata? - Cole
On Wed, Mar 23, 2016 at 12:09:29PM -0400, Cole Robinson wrote:
On 03/23/2016 11:34 AM, Kashyap Chamarthy wrote:
I'd imagine most developers on this list have their own workflows to track/apply/test patches from the mailing list. I normally just save patches in Mutt's Maildir and apply them manually to a Git branch. I recently began using the 'patches'[1] tool, that some of you on this list must be familiar with, to test arbitrary QEMU patches from the mailing list, and found it relatively easy large patch series from mailing list.
s/easy/easy to test/
I'm writing this to check if the libvirt upstream community finds it desirable to setup such a patches database.
I haven't played much with 'patches' so I can't give a first hand opinion. But if it's low effort to set up and maintain seems like a no-brainer to generate and publish the metadata so libvirt devs can start playing with it.
I haven't set up the database myself, I'm afraid. From my quick look at patches/patchlib/scan.py, it uses a `notmuch` database (an .xz conpressed file), and I just need to have a config file. I use `notmuch`[*] for email indexing sometimes, so I can try to play with this and setup a database. I was just a bit hesitant to start as I'll be away on leave for 2 weeks starting this weekend. Still, I'll try to see if I can squeeze this task in before I leave, since I brought it up.
Although your details don't cover it, one of the major benefits of 'patches' is that it also scoops up metadata like Reviewed-by:, and gives an easy way to get those tags into commit messages. Good for giving prolific reviewers credit (and shaming those that don't do enough reviews :) ).
:-) Understood. You can see the kind of prefixes it supports in the README.md.
The corollary is that you can search for unreviewed series, which IMO is desperately needed for libvir-list; the review time is pretty bad and I can't begin to count the number of drive-by patches we've received on list over the years that go completely unresponded. And I say this knowing full well I probably have the worst review-to-patches-authored ratio of all libvirt committers... but I don't live on libvir-list and often go 3-6 months at a time where I don't have any libvirt patches, and usually during those times I just skim-and-archive the list contents. If we had a programmatic tool to show me the unreviewed patches,
It shows something closer to unreviewed patches, looking at the terms supported by its query language (using `notmuch`; see the "Search Language" section in README.md): [...] - `status:rfc` show RFC postings - `status:committed` show committed series - `status:applied` show series with a "Thanks, applied" reply - `status:unapplied` short hand for `not (status:broken or status:obsolete or status:pull-request or status:rfc or status:committed or status:applied)` [...] - `status:reviewed` show series where every patch has at least one Reviewed-by - `reviewed-by:ADDRESS` show series if a patch has a Reviewed-by by `ADDRESS`
I could run that on occasion and pitch in much more easily, at least reviewing bits that don't need a lot of context. We could even have an informal policy like 'theres XXX number of unreviewed patches on the list... no @redhat people post patches until we review the backlog'
What if a patch series is NACKd or abandoned, is there a way to get it out of the patches metadata?
Seems like NACKed patches are displayed (I don't find any search term for "abandoned") can be fished out of the repo: [...] # I snipped out some of prefixes - `nacked-by:ADDRESS` show series if a patch has a Nacked-by by `ADDRESS` - `acked-by:ADDRESS` show series if a patch has a Acked-by by `ADDRESS` [*] https://notmuchmail.org/ -- /kashyap
On Wed, Mar 23, 2016 at 04:34:26PM +0100, Kashyap Chamarthy wrote:
I'd imagine most developers on this list have their own workflows to track/apply/test patches from the mailing list. I normally just save patches in Mutt's Maildir and apply them manually to a Git branch. I recently began using the 'patches'[1] tool, that some of you on this list must be familiar with, to test arbitrary QEMU patches from the mailing list, and found it relatively easy large patch series from mailing list.
I'm writing this to check if the libvirt upstream community finds it desirable to setup such a patches database.
It would probably be useful. I'll see if I can setup a cron job on the libvirt.org server to publish it and we can give it a try. If it works out, we might want to get a bit more formal in giving Reviewed-by lines, etc, rather than the generic ACK Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, Mar 23, 2016 at 04:40:48PM +0000, Daniel P. Berrange wrote:
On Wed, Mar 23, 2016 at 04:34:26PM +0100, Kashyap Chamarthy wrote:
I'd imagine most developers on this list have their own workflows to track/apply/test patches from the mailing list. I normally just save patches in Mutt's Maildir and apply them manually to a Git branch. I recently began using the 'patches'[1] tool, that some of you on this list must be familiar with, to test arbitrary QEMU patches from the mailing list, and found it relatively easy large patch series from mailing list.
I'm writing this to check if the libvirt upstream community finds it desirable to setup such a patches database.
It would probably be useful. I'll see if I can setup a cron job on the libvirt.org server to publish it and we can give it a try.
That'd be nice, if you have time. Indeed, the intention was to just try, informally.
If it works out, we might want to get a bit more formal in giving Reviewed-by lines, etc, rather than the generic ACK
Yep. -- /kashyap
On Wed, Mar 23, 2016 at 04:34:26PM +0100, Kashyap Chamarthy wrote: [snip] So things turn out to be a bit more complicated for libvirt than for QEMU, because we have one mailing list accepting patches for 10+ different git repositories. Patches has no built-in support for this model, so you have to create a separate configuration file for each repo, so that it stores the patches json file separately for each and then pass that config file when fetching git clone git://github.com/stefanha/patches.git REPOS="libvirt libvirt-designer libvirt-glib libvirt-java libvirt-perl libvirt-php libvirt-python libvirt-sandbox libvirt-snmp libvirt-tck" for repo in $REPOS do cat > $HOME/.patchesrc-$repo <<EOF [options] patches_dir=$HOME/.patches/$repo [fetch] url=http://libvirt.org/patches/$repo/patches.json EOF cat > $HOME/bin/patchesrc-$repo <<EOF #!/bin/sh exec $HOME/patches/patches --config $HOME/.patchesrc-$repo \$@ EOF chmod +x $HOME/bin/patchesrc-$repo done Now, assuming $HOME/bin is in your $PATH, you can run $ patches-libvirt fetch $ patches-libvirt list Or to look at other repos, eg sandbox $ patches-libvirt-sandbox fetch $ patches-libvirt-sandbox list patchesrc-libvirt-sandbox list Message-id: 20160323074705.GA4974@bogon.m.sigxcpu.org From: Guido Günther <agx@sigxcpu.org> Date: 2016-03-23 [1/1] Ignore generated file $ patches-libvirt-sandbox apply id:20160323074705.GA4974@bogon.m.sigxcpu.org nb assumes your $CWD is the libvirt-sandbox git checkout Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, Mar 23, 2016 at 06:46:53PM +0000, Daniel P. Berrange wrote:
On Wed, Mar 23, 2016 at 04:34:26PM +0100, Kashyap Chamarthy wrote: [snip]
So things turn out to be a bit more complicated for libvirt than for QEMU, because we have one mailing list accepting patches for 10+ different git repositories.
Ah-ha, good point.
Patches has no built-in support for this model, so you have to create a separate configuration file for each repo, so that it stores the patches json file separately for each and then pass that config file when fetching
git clone git://github.com/stefanha/patches.git
REPOS="libvirt libvirt-designer libvirt-glib libvirt-java libvirt-perl libvirt-php libvirt-python libvirt-sandbox libvirt-snmp libvirt-tck" for repo in $REPOS do cat > $HOME/.patchesrc-$repo <<EOF [options] patches_dir=$HOME/.patches/$repo
[fetch] url=http://libvirt.org/patches/$repo/patches.json EOF
Yeah, this nicely solves it currently, thanks for clearly spelling it out. How often are the JSON databases updated? Once daily at least, I presume.
cat > $HOME/bin/patchesrc-$repo <<EOF #!/bin/sh
exec $HOME/patches/patches --config $HOME/.patchesrc-$repo \$@ EOF
chmod +x $HOME/bin/patchesrc-$repo done
Now, assuming $HOME/bin is in your $PATH, you can run
$ patches-libvirt fetch $ patches-libvirt list
Or to look at other repos, eg sandbox
$ patches-libvirt-sandbox fetch $ patches-libvirt-sandbox list patchesrc-libvirt-sandbox list Message-id: 20160323074705.GA4974@bogon.m.sigxcpu.org From: Guido Günther <agx@sigxcpu.org> Date: 2016-03-23 [1/1] Ignore generated file
$ patches-libvirt-sandbox apply id:20160323074705.GA4974@bogon.m.sigxcpu.org
nb assumes your $CWD is the libvirt-sandbox git checkout
Excellent, I tested with just the libvirt repo, works perfectly well. Thanks for the quick action! -- /kashyap
On Thu, Mar 24, 2016 at 09:31:44AM +0100, Kashyap Chamarthy wrote:
On Wed, Mar 23, 2016 at 06:46:53PM +0000, Daniel P. Berrange wrote:
On Wed, Mar 23, 2016 at 04:34:26PM +0100, Kashyap Chamarthy wrote: [snip]
So things turn out to be a bit more complicated for libvirt than for QEMU, because we have one mailing list accepting patches for 10+ different git repositories.
Ah-ha, good point.
Patches has no built-in support for this model, so you have to create a separate configuration file for each repo, so that it stores the patches json file separately for each and then pass that config file when fetching
git clone git://github.com/stefanha/patches.git
REPOS="libvirt libvirt-designer libvirt-glib libvirt-java libvirt-perl libvirt-php libvirt-python libvirt-sandbox libvirt-snmp libvirt-tck" for repo in $REPOS do cat > $HOME/.patchesrc-$repo <<EOF [options] patches_dir=$HOME/.patches/$repo
[fetch] url=http://libvirt.org/patches/$repo/patches.json EOF
Yeah, this nicely solves it currently, thanks for clearly spelling it out.
How often are the JSON databases updated? Once daily at least, I presume.
Once an hour Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (3)
-
Cole Robinson -
Daniel P. Berrange -
Kashyap Chamarthy