Re: [libvirt] Libvirt upstream CI efforts

On 2/21/19 9:39 AM, Erik Skultety wrote: Nice to meet you lcitool! I spent some time looking and testing it, and I see tremendously value in allowing developers have the same experience locally (or anywhere else they choose) as opposed to only behind a black(-ish) box environment. Yash may remember some of our conversations about that. The problem lcitool solves is common (I'm having that myself for "deployment checks", AKA integration tests, of Avocado itself)[1]. Hopefully not diverting too much from the main topic, but I'd like to ask if there was a specific reason for installing guests instead of reusing something like virt-builder? This is my "provision" step that I use locally: $ virsh destroy $DOMAIN; virt-builder --ssh-inject=root:file:$SSH_PUB_KEY --selinux-relabel --root-password=password:$PASSWORD --output=$VM_BASE_DIR/$DOMAIN.qcow2 --format=qcow2 --install python2 $GUEST_TYPE && virsh start $DOMAIN Which seems to be quicker and simpler than maintaining kickstart files. It also covers more guests (should work for FreeBSD which seem to have some caveats on lcitool). Ideally, I'd like ansible to be responsible for it (and it'd be fine that it calls this or something else). But I haven't looked at how well ansible will take this (maybe a dynamic inventory implementation is all that's needed). This seems to match the virt-builder approach. Maybe this just needs to become a repository that developers can also download from? This would require the FreeBSD ISOs (and installation) to be converted into a similar pre-built image use, though. We're hosting a very small number of images (and small images in size) here: https://avocado-project.org/data/assets/ There's at least one image that gets downloaded on every single Avocado-VT installation (vt-bootstrap) by default. I have to admit I haven't monitored the bandwidth usage, but it hasn't gone over the quota (and we're paying ~5 USD/month for that server). One must pay attention to capabilities, seccomp and other layers added to containers. I'm not fully confident that the results of virtualization testing under a container (specially failures) are just as good as results from a non-containerized environment. But I may be on track to changing my opinion on this matter. Whenever you have an external test suite, you loose the automatic version matching of the component you're testing. Then conditionals, abstractions, special treatments for the components we're testing tend to plague everything. Avocado-VT/tp-{qemu,libvirt} are examples of test frameworks repositories that may still support 10 years or so of different software versions. The end result is *not* nice because: * Abstraction increases to support multiple versions of everything * Learning curve goes through the roof * Developers don't take the time to learn a complex framework full of abstractions * QE does take the time, because they usually need to more than one version of a software * Developers and QE now have their own silos You could overcome some of that by keeping policies on supported versions, baby sitting and deprecating code, but I firmly believe that those house keeping tasks are bound to fail. There's one thing developers will take immediate action on, and that is when a "make check[-functional]" fails... so a test suite in this sense need to be intrusive and affect a developer's common workflow. Well, "avocado-qemu" did not exist when we initially pursued this task. Besides the points above, as to why keeping the tests as part of the main repo, we understand that there are a lot of common problems in testing. They're usually solved over and over again, in an ad-hoc manner for each project. (Pure) Avocado was for a long time nothing but a speculation of what we believed most projects would need for their testing (and an Avocado-VT compatibility layer). We got somethings right, and somethings wrong. During the last year or so, a number of Avocado features have been added for the sake of QEMU testing (for the "avocado-qemu" initiative), but I bet that a user reading the documentation won't guess that. Those features are abstract and should work for any other project. So, along the way, we had confidence that the testing stack could be shared and split, and that tests living within the main repo ended up looking simple and effective. The "glue" between tests and framework is quite thin and the bootstrap can be done transparently as part of the "make check-acceptance" target. We haven't heard from developers any resistance to this approach, so, so far, we believe we're on the right track. Yes, this is the type of experience that should ultimately be delivered. [1] https://github.com/avocado-framework/avocado/tree/master/selftests/deploy... -- Cleber Rosa [ Sr Software Engineer - Virtualization Team - Red Hat ] [ Avocado Test Framework - avocado-framework.github.io ] [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]