Hi Michal, You asked how NVDIMM hotplug should work if the domain was launched without -M nvdimm=on. I'm not very familiar with ACPI so this may be incorrect. Hotplug relies on the presence of ACPI tables (SSDT Device=NVDR _HID=ACPI0012) added by -M nvdimm=on. The ACPI tables contain the interface needed to handle hotplug events. The device_add command fails with the following message if -M nvdimm=on is missing: nvdimm is not enabled: missing 'nvdimm' in '-M' I'm not sure if ACPI allows SSDT tables to be dynamically added but in any case the current QEMU code doesn't support that. Options for libvirt: 1. Rely on QEMU's device_add error. Pro: If QEMU behavior ever changes then libvirt will also start working without code changes. Cons: Does libvirt interpret QMP error strings?

2. Only allow NVDIMM hotplug if the domain was started with -M nvdimm=on. I think QEMU will not add -M nvdimm=on to the "pc" machine type by default since it adds the NVDIMM DSM hardware interface that increases the security attack surface. Stefan

BTW: I ran a migration from no nvdimm qemu to one that had -M nvdimm=on and guest migrated happily. So looks like guest ABI is stable (or at least stable enough not to crash). But since ACPI table is changed I doubt that.

Also, I found that hotunplug is not implemented yet?

error: internal error: unable to execute QEMU command 'device_del': nvdimm device hot unplug is not supported yet. Michal

On 09/14/2017 02:33 AM, Haozhong Zhang wrote: > On 09/13/17 17:28 +0200, Michal Privoznik wrote: >> >> BTW: I ran a migration from no nvdimm qemu to one that had -M nvdimm=on >> and guest migrated happily. So looks like guest ABI is stable (or at >> least stable enough not to crash). But since ACPI table is changed I >> doubt that. > > One example that may cause trouble is that > 1/ Guest OS got a pointer to an ACPI table A on the source host (w/o nvdimm=on) > 2/ After migrating to the destination host (w/ nvdimm=on), the > location of ACPI table A is occupied by NFIT. If guest OS tries to > access ACPI table A via the pointer in step 1/, then it will access > the wrong table. > Ah, you're right. So it a guest ABI breakage to add nvdimm=on. IOW, libvirt can't safely add that onto command line. Well we could for freshly started guest and not those which are just being migrated. But that increases attack surface so it's not safe either. Okay, I'll stick with the latest proposal (expose this in domain XML and let users turn it on if they want to).