On 22 Jul 2014, at 19:43, Alex Bligh <alex(a)alex.org.uk&gt; wrote: > Testing has been light to date (i.e. > can I migrate it inbound with -S without anything complaining).

I've given this quite a bit more testing today. It works fine qemu-kvm 1.0 -> qemu-2.0+patch (cirrus vga) It works fine qemu-2.0+patch -> qemu-2.0+patch (cirrus vga) It doesn't (yet) work qemu-2.0+patch -> qemu-kvm 1.0 (cirrus vga). The reason for this is (at least) that I need to emulate the broken versioning of the mc146818rtc timer section, as writing it correctly confuses qemu-kvm 1.0. Therefore please don't bother testing migration back to 1.0 yet.

Serge, > I don't think that is in any way a problem. Is migrating to older > versions ever actually expected to work? In either case I don't > think for this particular case it's a problem. Good; no; and good - respectively. > (The "how to handle this in libvirt" question is more interesting) I've been giving this some thought. However rococo we make this, I think the caller is often going to need to modify the command line anyway, i.e. is going to need to be aware of the migration problem. For instance, taking Ubuntu as an example, 12.04 shipped with qemu-kvm-1.0, and a pxe-virtio.rom of just under 64k, giving a 64k ROM slot. 14.04 ships with qemu-2.0 and a pxe-virtio.rom of over 80k, giving a 128k ROM slot. So however we fix the machine types, when migrating in a 12.04 initiated VM, qemu will need -global virtio-net-pci.romfile=/path/to/pxe-virtio.rom.12.04 on the command line (or, if you don't much care about PXE working on a soft reboot, a blank file of the same size), whereas when migrating in a 14.04 initiated VM, that must not be on the command line. Fixing this properly would entail requiring that the ROMs are (effectively) distributed with qemu or at least that all ROM sizes become part of the machine type standard. This would have the advantage that loading a larger ROM than the machine type specifies would fail unless the ROM size was explicitly configured on the command line. But this is a subject wider than this patch. So the long and the short of it is that libvirt (sadly) like anything else starting qemu machines needs to know a bit about different versions of qemu, and be able to replace a machine type option with a machine type option and more on the command line. My previous suggestion doesn't help much because qemu will still need to be passed something on the command line. I think the best way to go with this patch would be something like: * Add pc-1.0-qemu-kvm as a machine type (done) * Rename pc-1.0 to pc-1.0-qemu-git * Add an alias for pc-1.0 to either pc-1.0-qemu-git or pc-1.0-qemu-kvm, configurable at build time with a ./configure option. The distro can then set this appropriately. This would default to pc-1.0-qemu-git (i.e. the current behaviour). On distros that only every used one of the above, ./configure will sort things out, +/- self-inflicted injuries relating to ROM size changes. If life is more complicated, libvirt (and other callers) will need to be aware. pc-1.0-qemu-git and pc-1.0-qemu-kvm can be used to unambiguously mean the relevant machine type, which will fix things going forward for that machine type. WDYT?

And from there I think the thing to do will be to introduce a transient alternate package that has the pc-1.0 alias pointing ot pc-1.0-qemu-kvm

and depends on the legacy pxe rom.

Quoting Paolo Bonzini (pbonzini(a)redhat.com): > Il 29/07/2014 15:03, Serge E. Hallyn ha scritto: >> >> And from there I think the thing to do will be to introduce a transient >> alternate package that has the pc-1.0 alias pointing ot pc-1.0-qemu-kvm > > This should be done in the main package, too. That seems like a problem, unless I"m misunderstanding something. If we do that in the main package, then anyone running a pc-1.0 system under the qemu package won't be able to migrate. Wouldn't it be better to have pc-1.0 alias by default point to the pc-1.0-qemu machine type?

>> and depends on the legacy pxe rom. > > If you can make the pxe-virtio.rom file 64k or less, then that would be > a good idea for 14.04 in general. Newer machine types use > efi-virtio.rom, so you won't break "-M pc" migration. Hm. No idea offhand how I'd do that, but it sounds worth looking into.

> > I'm not sure either. You could simply package the 12.04 ipxe ROMs into > > 14.04, and add a note about getting the sources for GPL friendliness. This would be my preference (or in Ubuntu's case, add it to the ipxe-qemu package) but I think it should only be used when the legacy machine type is used or it will break inbound migrations from other 14.04 machines started with 128k ROMs (AIUI).

On 29 Jul 2014, at 14:42, Paolo Bonzini <pbonzini(a)redhat.com&gt; wrote: > Il 29/07/2014 15:39, Alex Bligh ha scritto: >>>> I'm not sure either. You could simply package the 12.04 ipxe ROMs into >>>> 14.04, and add a note about getting the sources for GPL friendliness. >> This would be my preference (or in Ubuntu's case, add it to the ipxe-qemu >> package) but I think it should only be used when the legacy machine type >> is used or it will break inbound migrations from other 14.04 machines >> started with 128k ROMs (AIUI). > > "-M pc" and its alias "-M pc-i440fx-2.0" doesn't use pxe-virtio.rom at > all (at least upstream). Really? How does it pxeboot on virtio then (it definitely does pxeboot on virtio)?

> Does Ubuntu 14.04 have efi-virtio.rom? Yes. After 2 layers of symlinks you get to. ubuntu@trustytest:~$ ls -la /usr/lib/ipxe/qemu/efi-virtio.rom -rw-r--r-- 1 root root 220672 Jan 6 2014 /usr/lib/ipxe/qemu/efi-virtio.rom > It would break "-M pc-1.0" started on older 14.04, but I think that's > acceptable. I was more worried about any previous versions of Ubuntu (newer than 12.04) which might also be using the larger rom size. But then I haven't investigated at what stage the rom size grew.

Il 29/07/2014 15:27, Serge Hallyn ha scritto: > Quoting Paolo Bonzini (pbonzini(a)redhat.com): >> Il 29/07/2014 15:03, Serge E. Hallyn ha scritto: >>> >>> And from there I think the thing to do will be to introduce a transient >>> alternate package that has the pc-1.0 alias pointing ot pc-1.0-qemu-kvm >> >> This should be done in the main package, too. > > That seems like a problem, unless I"m misunderstanding something. If we do > that in the main package, then anyone running a pc-1.0 system under the > qemu package won't be able to migrate. Wouldn't it be better to have > pc-1.0 alias by default point to the pc-1.0-qemu machine type? You'd break that for people who have already upgraded from 12.04 and 14.04 and are keeping the old machine type. You'd fix it for people who are upgrading now. I think providing a smoother upgrade path is worthwhile, even if it annoys someone else. Unfortunately the only solution is a lot of testing *before* a release, and in fact this is why 2.1 was delayed by a migration problem. Once the release is out, you'll have to make someone unhappy.

>>> and depends on the legacy pxe rom. >> >> If you can make the pxe-virtio.rom file 64k or less, then that would be >> a good idea for 14.04 in general. Newer machine types use >> efi-virtio.rom, so you won't break "-M pc" migration. > > Hm. No idea offhand how I'd do that, but it sounds worth looking into. I'm not sure either. You could simply package the 12.04 ipxe ROMs into 14.04, and add a note about getting the sources for GPL friendliness.

On 29 Jul 2014, at 14:21, Paolo Bonzini <pbonzini(a)redhat.com&gt; wrote: > If you can make the pxe-virtio.rom file 64k or less, then that would be > a good idea for 14.04 in general. Newer machine types use > efi-virtio.rom, so you won't break "-M pc" migration. Without further, won't that break migration from 14.04 *with* the big ROM?

On Tue, Jul 29, 2014 at 08:31:28AM +0100, Alex Bligh wrote: > Serge, > > > I don't think that is in any way a problem. Is migrating to older > > versions ever actually expected to work? In either case I don't > > think for this particular case it's a problem. > > Good; no; and good - respectively. > > > (The "how to handle this in libvirt" question is more interesting) > > I've been giving this some thought. However rococo we make this, > I think the caller is often going to need to modify the command > line anyway, i.e. is going to need to be aware of the migration > problem. > > For instance, taking Ubuntu as an example, 12.04 shipped with > qemu-kvm-1.0, and a pxe-virtio.rom of just under 64k, giving > a 64k ROM slot. 14.04 ships with qemu-2.0 and a pxe-virtio.rom > of over 80k, giving a 128k ROM slot. So however we fix the > machine types, when migrating in a 12.04 initiated VM, qemu > will need > -global virtio-net-pci.romfile=/path/to/pxe-virtio.rom.12.04 > on the command line (or, if you don't much care about PXE > working on a soft reboot, a blank file of the same size), > whereas when migrating in a 14.04 initiated VM, that must > not be on the command line. > > Fixing this properly would entail requiring that the ROMs are > (effectively) distributed with qemu or at least that all > ROM sizes become part of the machine type standard. This > would have the advantage that loading a larger ROM than > the machine type specifies would fail unless the ROM > size was explicitly configured on the command line. But > this is a subject wider than this patch. > > So the long and the short of it is that libvirt (sadly) like > anything else starting qemu machines needs to know a bit about > different versions of qemu, and be able to replace a machine > type option with a machine type option and more on the > command line. > > My previous suggestion doesn't help much because qemu will > still need to be passed something on the command line. > > I think the best way to go with this patch would be something > like: > > * Add pc-1.0-qemu-kvm as a machine type (done) > > * Rename pc-1.0 to pc-1.0-qemu-git > > * Add an alias for pc-1.0 to either pc-1.0-qemu-git or > pc-1.0-qemu-kvm, configurable at build time with > a ./configure option. The distro can then set this > appropriately. This would default to pc-1.0-qemu-git > (i.e. the current behaviour). > > On distros that only every used one of the above, > ./configure will sort things out, +/- self-inflicted > injuries relating to ROM size changes. If life is > more complicated, libvirt (and other callers) will > need to be aware. pc-1.0-qemu-git and pc-1.0-qemu-kvm > can be used to unambiguously mean the relevant machine > type, which will fix things going forward for that > machine type. > > WDYT? This just means we perpetuate the broken-ness. I would rather we teach libvirt to do the right thing unconditionally.

On Mon, Aug 04, 2014 at 03:08:31PM +0000, Serge Hallyn wrote: > Quoting Michael S. Tsirkin (mst(a)redhat.com): > > On Tue, Jul 29, 2014 at 08:31:28AM +0100, Alex Bligh wrote: > > > Serge, > > > > > > > I don't think that is in any way a problem. Is migrating to older > > > > versions ever actually expected to work? In either case I don't > > > > think for this particular case it's a problem. > > > > > > Good; no; and good - respectively. > > > > > > > (The "how to handle this in libvirt" question is more interesting) > > > > > > I've been giving this some thought. However rococo we make this, > > > I think the caller is often going to need to modify the command > > > line anyway, i.e. is going to need to be aware of the migration > > > problem. > > > > > > For instance, taking Ubuntu as an example, 12.04 shipped with > > > qemu-kvm-1.0, and a pxe-virtio.rom of just under 64k, giving > > > a 64k ROM slot. 14.04 ships with qemu-2.0 and a pxe-virtio.rom > > > of over 80k, giving a 128k ROM slot. So however we fix the > > > machine types, when migrating in a 12.04 initiated VM, qemu > > > will need > > > -global virtio-net-pci.romfile=/path/to/pxe-virtio.rom.12.04 > > > on the command line (or, if you don't much care about PXE > > > working on a soft reboot, a blank file of the same size), > > > whereas when migrating in a 14.04 initiated VM, that must > > > not be on the command line. > > > > > > Fixing this properly would entail requiring that the ROMs are > > > (effectively) distributed with qemu or at least that all > > > ROM sizes become part of the machine type standard. This > > > would have the advantage that loading a larger ROM than > > > the machine type specifies would fail unless the ROM > > > size was explicitly configured on the command line. But > > > this is a subject wider than this patch. > > > > > > So the long and the short of it is that libvirt (sadly) like > > > anything else starting qemu machines needs to know a bit about > > > different versions of qemu, and be able to replace a machine > > > type option with a machine type option and more on the > > > command line. > > > > > > My previous suggestion doesn't help much because qemu will > > > still need to be passed something on the command line. > > > > > > I think the best way to go with this patch would be something > > > like: > > > > > > * Add pc-1.0-qemu-kvm as a machine type (done) > > > > > > * Rename pc-1.0 to pc-1.0-qemu-git > > > > > > * Add an alias for pc-1.0 to either pc-1.0-qemu-git or > > > pc-1.0-qemu-kvm, configurable at build time with > > > a ./configure option. The distro can then set this > > > appropriately. This would default to pc-1.0-qemu-git > > > (i.e. the current behaviour). > > > > > > On distros that only every used one of the above, > > > ./configure will sort things out, +/- self-inflicted > > > injuries relating to ROM size changes. If life is > > > more complicated, libvirt (and other callers) will > > > need to be aware. pc-1.0-qemu-git and pc-1.0-qemu-kvm > > > can be used to unambiguously mean the relevant machine > > > type, which will fix things going forward for that > > > machine type. > > > > > > WDYT? > > > > > > This just means we perpetuate the broken-ness. > > > > I would rather we teach libvirt to do the right thing > > unconditionally. > > Well, now, here's a thought - can we hot-patch qemu to > change the machine type while it is running before the > migrate? Just s/pc-1.0/pc-x.0/ or something? Frankly I don't see what will this accomplish. If you really want it to be called pc-1.0, you can make it a machine property instead. E.g. qemu-kvm-compatibility. Teach management to set it if remote is qemu-kvm: -machine pc-1.0,qemu-kvm-compatibility=on

On 4 Aug 2014, at 16:38, Serge Hallyn <serge.hallyn(a)ubuntu.com&gt; wrote: >> >> If you really want it to be called pc-1.0, you >> can make it a machine property instead. >> E.g. qemu-kvm-compatibility. >> Teach management to set it if remote is qemu-kvm: >> -machine pc-1.0,qemu-kvm-compatibility=on > > That sounds nice - Alex, what do you think? Not having used the machine property stuff before, or played with libvirt much, I'm not sure how this helps libvirt. I thought the issue here was that migrating from 1.0-qemu-kvm to 2.x OR 1.0-qemu-git to 2.x, libvirt is going to to supply the same command line. As libvirt doesn't know what the sender is (and it's not possible to detect this automatically - at least not without a far more intrusive patch),

one has to make a choice at build time as to what 'pc-1.0' represents.

This is what patch #2 does. I fully agree it is not pretty.

So I am not sure why -machine pc-1.0,qemu-kvm-compatibility=on is any easier for libvirt than -machine pc-1.0-qemu-kvm IE what does using a machine property rather than a machine type buy us?

-- Alex Bligh