On 11/30/2015 06:05 AM, Boris Fiuczynski wrote: > This patch reverts a part of commit 0d8b24f6b. > With commits 0d8b24g6 and 0785966d automatically adding a required controller g6 ? I assume you meant f6b...

> had been moved from the domain xml parsing code section into the device post > xml parsing code section and in doing so breaking as a side effect the SCSI > hostdev hotplugging in case the required SCSI controller had not been defined > in the domain. What side effect?

processing?

> This behavior occurs because the SCSI controller gets added but > is NOT hotplugged. In the hotplug code section the controller is searched for > and if not found would be hotplugged but since a SCSI controller already exists > in the domain defintion the required SCSI controller hotplug is never executed. s/defintion/definition

Hmmm, is this the side effect? It perhaps would have been helpful to know what API in the hotplug section of code you're referencing. Are you referencing qemuDomainFindOrCreateSCSIDiskController?

> This results later in the internal error: > > error: Failed to attach device from st0.xml > error: internal error: Device alias was not set for scsi controller with index 0 > I can reproduce this issue if I define/start a domain with no SCSI controllers. Then whether the added hostdev has a drive address or not, the hotplug fails. I also get the same results for disk hotplug, so perhaps the issue needs a more "generic" solution.

> This patch moves the automatic add of a SCSI controller for a SCSI hostdev > device back into the domain xml parsing code section. > I'm still baffled why this "works", but need some time to work through the algorithms.

> Signed-off-by: Boris Fiuczynski <fiuczy(a)linux.vnet.ibm.com&gt; > Reviewed-by: Bjoern Walk <bwalk(a)linux.vnet.ibm.com&gt; > Reviewed-by: Stefan Zimmermann <stzi(a)linux.vnet.ibm.com&gt; > --- > src/conf/domain_conf.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c > index cbfc41e..69cfd0f 100644 > --- a/src/conf/domain_conf.c > +++ b/src/conf/domain_conf.c > @@ -16285,6 +16285,9 @@ virDomainDefParseXML(xmlDocPtr xml, > } > > def->hostdevs[def->nhostdevs++] = hostdev; > + > + if (virDomainDefMaybeAddHostdevSCSIcontroller(def) < 0) > + goto error; What if the hotplug <hostdev> xml provided an address using perhaps controller==1? I think this may only work in the default case and/or when controller==0 in the provided an address case. If we were to go ahead with this patch, I'd have to merge in patch2 as well. That way a git bisect that falls in between patch 1 and 2 won't cause some "other" issue...

I'm still trying to "internalize" the whole issue, this is where I'm at. Prior to any of my changes, at parse processing if a <hostdev> didn't have an <address> element, it would be added. Then nhostdevs would be incremented, and virDomainDefMaybeAddHostdevSCSIcontroller would be called. That code may call virDomainDefMaybeAddController or return 0 (and not call virDomainDefMaybeAddController). With my changes, address assignment is deferred to post processing as well as maybe adding a controller. At post processing, for a scsi hostdev without an address defined, virDomainHostdevAssignAddress is called which would call virDomainDefMaybeAddController if one wasn't found or one was found, but was full. That would seemingly add the controller via virDomainDefMaybeAddController as would the virDomainDefMaybeAddHostdevSCSIcontroller. But it seems perhaps for some reason it shouldn't or didn't. Let's say it didn't add it (for whatever reason), it seems the expectation is that when qemuDomainFindOrCreateSCSIDiskController is called it won't find the controller present and create/add it, which I think is the side effect referenced in the commit message. I think I'm just missing some nuance, but I'll keep poking to figure it out. I at least wanted to provide some feedback to ensure this wasn't reviewed/pushed by someone else! Although if someone else is looking at it and has some ideas that'd be great too. John > } > VIR_FREE(nodes); > >

On 11/30/2015 06:05 AM, Boris Fiuczynski wrote: [...] > diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c > index cbfc41e..69cfd0f 100644 > --- a/src/conf/domain_conf.c > +++ b/src/conf/domain_conf.c > @@ -16285,6 +16285,9 @@ virDomainDefParseXML(xmlDocPtr xml, > } > > def->hostdevs[def->nhostdevs++] = hostdev; > + > + if (virDomainDefMaybeAddHostdevSCSIcontroller(def) < 0) > + goto error; > } > VIR_FREE(nodes); > > After some more experimentation and making sure my debug environment was "up to date" (e.g. had virtlogd started)... Using top of tree I can hotplug a hostdev as long as it has a drive address defined. Not sure why it was failing before, but at least things are making more sense now. The reason why that path works with top of tree is because virDomainDeviceDefPostParseInternal won't call virDomainHostdevAssignAddress since hdev->info->type is not VIR_DOMAIN_DEVICE_ADDRESS_TYPE_NONE. That means when the qemuDomainFindOrCreateSCSIDiskController is called it will do the magic add. The path for this patch handles the inactive domain definition. While patch 2 handles the backend of the active domain path.

When you have a running domain, then qemuDomainAttachDeviceFlags (e.g. what virsh attach-device would use) calls virDomainDeviceDefParse which is where the controller is added during virDomainDeviceDefPostParse. So patch 2 where you're removing that PostParse callback is why the hotplug will then work because no longer does post parse add the controller to the domain def.

The call to qemuDomainFindOrCreateSCSIDiskController called by qemuDomainAttachSCSIDisk during qemuDomainAttachDeviceDiskLive in the live path of qemuDomainAttachDeviceFlags would then automagically add the controller. Since the active path wouldn't have added it during post parse.

While these patches would work for the active domain hostdev add, I think perhaps a different mechanism should be used. Given the failure from trying to add a disk in the same environment, I'm not sure either patch solves the root problem that during hotplug we're relying on *not* adding a controller device when a new one is deemed necessary (for both hostdev and disk).

One thing that is of interest is that virDomainDefMaybeAddController has 3 return values -1 fail, 0 didn't add a controller, and 1 added a controller. I'm wondering if that can somehow be used to ensure that if we add a controller that we ensure it really gets plugged in. Instead of relying on qemuDomainFindOrCreateSCSIDiskController to backdoor it in.

John

On 12/02/2015 03:27 AM, Boris Fiuczynski wrote: > On 12/02/2015 01:16 AM, John Ferlan wrote: >> >> >> On 11/30/2015 06:05 AM, Boris Fiuczynski wrote: >> >> [...] >> >>> diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c >>> index cbfc41e..69cfd0f 100644 >>> --- a/src/conf/domain_conf.c >>> +++ b/src/conf/domain_conf.c >>> @@ -16285,6 +16285,9 @@ virDomainDefParseXML(xmlDocPtr xml, >>> } >>> >>> def->hostdevs[def->nhostdevs++] = hostdev; >>> + >>> + if (virDomainDefMaybeAddHostdevSCSIcontroller(def) < 0) >>> + goto error; >>> } >>> VIR_FREE(nodes); >>> >>> >> >> After some more experimentation and making sure my debug environment was >> "up to date" (e.g. had virtlogd started)... >> >> Using top of tree I can hotplug a hostdev as long as it has a drive >> address defined. Not sure why it was failing before, but at least things >> are making more sense now. The reason why that path works with top of >> tree is because virDomainDeviceDefPostParseInternal won't call >> virDomainHostdevAssignAddress since hdev->info->type is not >> VIR_DOMAIN_DEVICE_ADDRESS_TYPE_NONE. That means when the >> qemuDomainFindOrCreateSCSIDiskController is called it will do the >> magic add. >> >> The path for this patch handles the inactive domain definition. While >> patch 2 handles the backend of the active domain path. > Patch 2 is also used in the "inactive domain definition" case. > The important difference is that the controller is NOT created on the > device definition path so that the controller is added via hotplug by > qemuDomainFindOrCreateSCSIDiskController since device hotplug uses the > device definition path! That is why this patch reintroduces adding the > controller into domain definition that the hotplug code does not use. > >> >> When you have a running domain, then qemuDomainAttachDeviceFlags (e.g. >> what virsh attach-device would use) calls virDomainDeviceDefParse which >> is where the controller is added during virDomainDeviceDefPostParse. So >> patch 2 where you're removing that PostParse callback is why the hotplug >> will then work because no longer does post parse add the controller to >> the domain def. > Correct. > >> >> The call to qemuDomainFindOrCreateSCSIDiskController called by >> qemuDomainAttachSCSIDisk during qemuDomainAttachDeviceDiskLive in the >> live path of qemuDomainAttachDeviceFlags would then automagically add >> the controller. Since the active path wouldn't have added it during post >> parse. > Correct. > >> >> While these patches would work for the active domain hostdev add, I >> think perhaps a different mechanism should be used. Given the failure >> from trying to add a disk in the same environment, I'm not sure either >> patch solves the root problem that during hotplug we're relying on *not* >> adding a controller device when a new one is deemed necessary (for both >> hostdev and disk). > I think that the root problem was introduced by moving the adding of the > controller from the DOMAIN definition parsing into the DEVICE definition > parsing. It's all domain_conf code... virDomainHostdevDefParseXML is either called by virDomainDefParseXML (inactive) or virDomainDeviceDefParse (active, via various drivers). The post parse processing code for driver callbacks is handled more directly through virDomainDeviceDefPostParse at the end of virDomainDeviceDefParse. The post parse processing for inactive callbacks is handled through virDomainDefPostParseDeviceIterator as part of validating the domain in virDomainDefPostParse. IMO it's splitting hairs to claim device processing is that much different than domain processing. It's just that the hotplug code knows how to directly go to the device it wants based on the xml from the attach-device command rather than going through all domain processing. In any case, the existing mechanism to assign an address and thus maybe add a controller for the inactive domain was introduced by commit id 'cdb978955'. That also introduced the code used by the active/hotplug processing. The "side effect" of the code not trying to add a controller was that setting the controller value to something didn't exist (I assume) knowing that CreateSCSIDiskController would do the dirty work later (not documented either which is another "issue" in my mind). Just for completeness... Auto-generating a controller for a hostdev was introduced by commit id '881eb780'. The CreateSCSIDiskController code has been around a lot longer... > Since the code at this point seems by its multiple usage patterns rather > complex and fragile to changes I choose for this fix to revert the code > rather than to reengineer the code and not potentially make things worse. > I think this may be a short term solution unless I can figure out a better way short term to hotplug an added controller as well if one is created. >> >> One thing that is of interest is that virDomainDefMaybeAddController has >> 3 return values -1 fail, 0 didn't add a controller, and 1 added a >> controller. I'm wondering if that can somehow be used to ensure that if >> we add a controller that we ensure it really gets plugged in. Instead of >> relying on qemuDomainFindOrCreateSCSIDiskController to backdoor it in. > I must say that calling vir > DomainDef < MaybeAddController from the > DEVICE parsing and not from the DOMAIN parsing feels rather strange to > me. In both cases domain parsing and hotplugging devices the controllers > are plugged in AFTER the device parsing is finished! I do not see any > backdoor on the hotplug path! > The "concept" of moving virDomainDefMaybeAddController into virDomainHostdevAssignAddress was I believe meant to be an optimization of sorts, but yes, it's turned into a bug or exposure of one of those unwritten/undocumented code paths which "expected" to have some other code path do the add. As for a different way to fix the problem... When a controller is really added into the domain it will gain an alias (which is what those error messages are complaining about). So, what if qemuDomainFindOrCreateSCSIDiskController checked the "found" cont in the first loop for "cont->info.alias". If it doesn't exist, then fall through under the "assumption" that the controller was added by some other mechanism into the list? That would require some more adjustments in qemuDomainAttachControllerDevice to know the controller was already in the list, but not active in the domain.

John

On 11/30/2015 06:06 AM, Boris Fiuczynski wrote: > When a SCSI disk is hotplugged to a domain that does not have the required > scsi controller already defined the following internal error occurs > > error: Failed to attach device from scsi_disk.xml > error: internal error: Could not find scsi controller with index 0 required for device > > Commit 0260506c added in method qemuBuildDriveDevStr a lookup of the controller > alias. The internal error occurs because in method qemuDomainAttachSCSIDisk > the automatic creation of the potentially missing SCSI controller occurs after > calling qemuBuildDriveDevStr. > > This patch reverses the calling sequence. > > Signed-off-by: Boris Fiuczynski <fiuczy(a)linux.vnet.ibm.com&gt; > Reviewed-by: Bjoern Walk <bwalk(a)linux.vnet.ibm.com&gt; > Reviewed-by: Stefan Zimmermann <stzi(a)linux.vnet.ibm.com&gt; > --- > src/qemu/qemu_hotplug.c | 12 ++++++------ > 1 file changed, 6 insertions(+), 6 deletions(-) > > diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c > index 8804d3d..210d485 100644 > --- a/src/qemu/qemu_hotplug.c > +++ b/src/qemu/qemu_hotplug.c > @@ -607,6 +607,12 @@ qemuDomainAttachSCSIDisk(virConnectPtr conn, > goto error; > } Would you say the following is an accurate description to add? /* Let's make sure our disk has a controller defined and loaded * before trying add the disk. The virDomainDiskDefAssignAddress * doesn't try to add a controller if one doesn't exist, it just * assigns the disk to the calculated spot. */

> > + for (i = 0; i <= disk->info.addr.drive.controller; i++) { > + cont = qemuDomainFindOrCreateSCSIDiskController(driver, vm, i); > + if (!cont) > + goto error; > + } > + > if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_DEVICE)) { > if (qemuAssignDeviceDiskAlias(vm->def, disk, priv->qemuCaps) < 0) > goto error; > @@ -617,12 +623,6 @@ qemuDomainAttachSCSIDisk(virConnectPtr conn, > if (!(drivestr = qemuBuildDriveStr(conn, disk, false, priv->qemuCaps))) > goto error; > > - for (i = 0; i <= disk->info.addr.drive.controller; i++) { > - cont = qemuDomainFindOrCreateSCSIDiskController(driver, vm, i); > - if (!cont) > - goto error; > - } > - > /* Tell clang that "cont" is non-NULL. > This is because disk->info.addr.driver.controller is unsigned, > and hence the above loop must iterate at least once. */ > Is there a reason you chose to not grab the clang check too?

John