Re: [libvirt] [PATCHv10 0/4] Introduce x86 Cache Monitoring Technology (CMT)

Hi John, The issue mentioned is generated by a -1 returned by qemuDomainGetIOThreadsMon. Further, it is caused by a -1 returned by virQEMUCapsGet: 6174 if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_OBJECT_IOTHREAD)) { 6175 virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", 6176 _("IOThreads not supported with this binary")); 6177 goto endjob; 6178 } By checking qemuCaps->flags, it seems the bit of QEMU_CAPS_OBJECT_IOTHREAD(176) is not set. (gdb) call virBitmapFormat(qemuCaps->flags) $4 = 0x7f08a4004cb0 "13,33,46,50,54-55,58,61-64,66-70,72-73,77-78,80,87-88,92-93,95-97,99,102-110,112,114,119-121,128-131,141-142,146,148-149,151-156,158-161,165,167,174-175,180,182,188,193-198,210,214,221-222,225,250,255"...

Breakpoint 1 at 0x7f088ef00880: file qemu/qemu_driver.c, line 5492. (gdb) c Continuing. [Switching to Thread 0x7f08bcdcb700 (LWP 24711)] Breakpoint 1, qemuDomainGetIOThreadsMon (driver=0x7f0874049b80, vm=0x7f08740cd840, iothreads=0x7f08bcdca8b8) at qemu/qemu_driver.c:5492 5492 { Unprocessed breakpoint [1] (gdb) finish Run till exit from #0 qemuDomainGetIOThreadsMon (driver=0x7f0874049b80, vm=0x7f08740cd840, iothreads=0x7f08bcdca8b8) at qemu/qemu_driver.c:5492 0x00007f088ef0098f in qemuDomainGetStatsIOThread (driver=<optimized out>, dom=<optimized out>, record=0x7f08ac000b60, maxparams=0x7f08bcdca9e4, privflags=<optimized out>) at qemu/qemu_driver.c:20885 20885 if ((niothreads = qemuDomainGetIOThreadsMon(driver, dom, &iothreads)) < 0) Value returned is $1 = -1 (gdb) bt #0 0x00007f088ef0098f in qemuDomainGetStatsIOThread (driver=<optimized out>, dom=<optimized out>, record=0x7f08ac000b60, maxparams=0x7f08bcdca9e4, privflags=<optimized out>) at qemu/qemu_driver.c:20885 #1 0x00007f088eee70a9 in qemuDomainGetStats (flags=1, record=<synthetic pointer>, stats=255, dom=0x7f08740cd840, conn=0x7f08ac000c50) at qemu/qemu_driver.c:21062 #2 qemuConnectGetAllDomainStats (conn=0x7f08ac000c50, doms=<optimized out>, ndoms=<optimized out>, stats=255, retStats=0x7f08bcdcab10, flags=<optimized out>) at qemu/qemu_driver.c:21162 #3 0x00007f08cd147646 in virConnectGetAllDomainStats (conn=0x7f08ac000c50, stats=0, retStats=retStats@entry=0x7f08bcdcab10, flags=0) at libvirt-domain.c:11685 #4 0x0000562e2404d2c0 in remoteDispatchConnectGetAllDomainStats (server=0x562e24ee74a0, msg=<optimized out>, ret=0x7f08ac001ec0, args=0x7f08ac001f30, rerr=0x7f08bcdcac50, client=0x562e24ef5200) at remote/remote_daemon_dispatch.c:6665 #5 remoteDispatchConnectGetAllDomainStatsHelper (server=0x562e24ee74a0, client=0x562e24ef5200, msg=<optimized out>, rerr=0x7f08bcdcac50, args=0x7f08ac001f30, ret=0x7f08ac001ec0) at remote/remote_daemon_dispatch_stubs.h:743 #6 0x00007f08cd05dad5 in virNetServerProgramDispatchCall (msg=0x562e24ef5910, client=0x562e24ef5200, server=0x562e24ee74a0, prog=0x562e24ee6650) at rpc/virnetserverprogram.c:437 #7 virNetServerProgramDispatch (prog=0x562e24ee6650, server=server@entry=0x562e24ee74a0, client=0x562e24ef5200, msg=0x562e24ef5910) at rpc/virnetserverprogram.c:304 #8 0x00007f08cd063f7d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x562e24ee74a0) at rpc/virnetserver.c:144 #9 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x562e24ee74a0) at rpc/virnetserver.c:165 #10 0x00007f08ccf95171 in virThreadPoolWorker (opaque=opaque@entry=0x562e24ee69a0) at util/virthreadpool.c:167 #11 0x00007f08ccf944f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #12 0x00007f08cb213dc5 in start_thread (arg=0x7f08bcdcb700) at pthread_create.c:308 #13 0x00007f08cab3b76d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 (gdb) My domain configuration is： [root@dl-c200 libvirt]# virsh list v^H Id Name State ---------------------------------- 1 ubuntu16.04-base running [root@dl-c200 libvirt]# virsh dumpxml 1 <domain type='kvm' id='1'> <name>ubuntu16.04-base</name> <uuid>19d01bd1-ad54-4176-84bd-e510de08fe00</uuid> <memory unit='KiB'>10485760</memory> <currentMemory unit='KiB'>10485760</currentMemory> <vcpu placement='static' current='4'>8</vcpu> <cputune> <cachetune vcpus='0' id='vcpus_0'> <monitor level='3' vcpus='0'/> </cachetune> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Broadwell</model> <feature policy='require' name='hypervisor'/> <feature policy='require' name='xsaveopt'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/tmp/kvm-donot-delete/vm-ubuntu16.04-base'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu'/> <target dev='hda' bus='ide'/> <readonly/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <mac address='52:54:00:eb:24:d3'/> <source bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <graphics type='vnc' port='5901' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c157,c912</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c157,c912</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain> I wonder if these message is enough for you to locate the root cause, I don’t have too much time in investigating this today, I'll try to create a fix tomorrow if you need. Huaqiang > -----Original Message----- > From: Wang, Huaqiang > Sent: Tuesday, November 27, 2018 11:21 AM > To: 'John Ferlan' <jferlan(a)redhat.com>; libvir-list(a)redhat.com > Subject: RE: [PATCHv10 0/4] Introduce x86 Cache Monitoring Technology > (CMT) > > > >> -----Original Message----- >> From: John Ferlan [mailto:jferlan@redhat.com] >> Sent: Tuesday, November 27, 2018 10:59 AM >> To: Wang, Huaqiang <huaqiang.wang(a)intel.com>; libvir-list(a)redhat.com >> Subject: Re: [PATCHv10 0/4] Introduce x86 Cache Monitoring Technology >> (CMT) >> >> >> >> On 11/26/18 9:39 PM, Wang, Huaqiang wrote: >>> Hi John, >>> >>> Really appreciate your hard work for the CMT series. Next I'll >>> working on >> the MBM. >>> >>> In testing the newly pushed code, I find a problem: >>> >>> <error message> >>> [david@dl-c200 ~]$ sudo virsh domstats >>> error: An error occurred, but the cause is unknown </error message> >>> >> >> I couldn't reproduced in a quick test here. Can you get a thread trace >> of the failure? >> >> What I usually do, build libvirt, then in a terminal session at the >> top of the git tree "./run gdb src/libvirtd" (dbg> r)... THen in >> another terminal session run the virsh command and when the libvirtd > session stops do a "t a a bt" >> (thread apply all backtrace)... >> >> John > > I'll trace the error. > > Thanks. > Huaqiang > >> >> (done for the night) >> >>> seems it is caused by qemuDomainGetStatsIOThread not by the new >> series. >>> What I test is return 0 immediately at top of >>> qemuDomainGetStatsIOThread, the command 'virsh domstats' reports >> the cache statistics normally. >>> >>> BR >>> Huaqiang >>> >>>> -----Original Message----- >>>> From: John Ferlan [mailto:jferlan@redhat.com] >>>> Sent: Tuesday, November 27, 2018 9:49 AM >>>> To: Wang, Huaqiang <huaqiang.wang(a)intel.com>; libvir- >> list(a)redhat.com >>>> Subject: Re: [PATCHv10 0/4] Introduce x86 Cache Monitoring >> Technology >>>> (CMT) >>>> >>>> >>>> >>>> On 11/26/18 12:56 PM, Wang Huaqiang wrote: >>>>> These patches are the remaining part for the CMT enabling series, >>>>> and most of the series have been merged. >>>>> >>>>> This series is addressing John's review comments and suggestions, >>>>> which are >>>>> https://www.redhat.com/archives/libvir-list/2018- >>>> November/msg00907.htm >>>>> l >>>>> https://www.redhat.com/archives/libvir-list/2018- >>>> November/msg00510.htm >>>>> l >>>>> https://www.redhat.com/archives/libvir-list/2018- >>>> November/msg00561.htm >>>>> l >>>>> >>>>> Change lists: >>>>> Changes in v10: >>>>> -. Add tag (virResctrlMonitorType) in > qemuDomainGetResctrlMonData, >>>> thus >>>>> qemuDomainGetResctrlMonData could be reused for MBM. >>>>> -. Using VIR_APPEND_ELEMENT to append > virQEMUResctrlMonDataPtr >>>> list. >>>>> -. Add qemuDomainFreeResctrlMonData. >>>>> -. Add virResctrlMonitorFreeStats. >>>>> -. Return a list of virResctrlMonitorStatsPtr instead of >>>>> a virResctrlMonitorStats array in virResctrlMonitorGetStats. >>>>> >>>>> Changes in V9: >>>>> -. Addressing code review comments form John. >>>>> -. Refined the names for new data structure and new functions. >>>>> -. Merged qemuDomainGetStatsCpuResMonitorPerTag and >>>> qemuDomainGetStatsCpuResMonitor, >>>>> and refined new function name based on the fact that we only >>>>> support >>>> cache monitor now. >>>>> Wang Huaqiang (4): >>>>> util: Return a list of pointer in virResctrlMonitorGetStats >>>>> util: Add function to free monitor statistical data >>>>> qemu: Report cache occupancy (CMT) with domstats >>>>> docs: Updated news.xml for CMT >>>>> >>>>> docs/news.xml | 12 ++++ >>>>> src/libvirt-domain.c | 12 ++++ >>>>> src/libvirt_private.syms | 1 + >>>>> src/qemu/qemu_driver.c | 183 >>>> ++++++++++++++++++++++++++++++++++++++++++++++- >>>>> src/util/virresctrl.c | 26 +++++-- >>>>> src/util/virresctrl.h | 8 ++- >>>>> tools/virsh.pod | 14 ++++ >>>>> 7 files changed, 248 insertions(+), 8 deletions(-) >>>>> >>>> >>>> Reviewed-by: John Ferlan <jferlan(a)redhat.com&gt; >>>> (series) >>>> >>>> and pushed, >>>> >>>> John