On 10/12/19 11:07 AM, Han Han wrote:
>
>
> On Sat, Oct 12, 2019 at 1:05 AM Cole Robinson <crobinso@redhat.com
> <mailto:crobinso@redhat.com>> wrote:
>
>     On 10/10/19 11:25 PM, Han Han wrote:
>     > Hi Cole,
>     > I merged crobinso/qcow2-data_file branch to 37b565c00. Reserved new
>     > capabilities introduced by these to branches to resolve conflicts.
>     > Then build and test as following:
>     > # ./autogen.sh&& ./configure --without-libssh
>     > --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu
>     > --program-prefix= --disable-dependency-tracking --prefix=/usr
>     > --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin
>     > --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include
>     > --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var
>     > --sharedstatedir=/var/lib --mandir=/usr/share/man
>     > --infodir=/usr/share/info --with-qemu --without-openvz --without-lxc
>     > --without-vbox --without-libxl --with-sasl --with-polkit
>     --with-libvirtd
>     > --without-phyp --with-esx --without-hyperv --without-vmware
>     > --without-xenapi --without-vz --without-bhyve --with-interface
>     > --with-network --with-storage-fs --with-storage-lvm
>     --with-storage-iscsi
>     > --with-storage-iscsi-direct --with-storage-scsi --with-storage-disk
>     > --with-storage-mpath --with-storage-rbd --without-storage-sheepdog
>     > --with-storage-gluster --without-storage-zfs
>     --without-storage-vstorage
>     > --with-numactl --with-numad --with-capng --without-fuse --with-netcf
>     > --with-selinux --with-selinux-mount=/sys/fs/selinux
>     --without-apparmor
>     > --without-hal --with-udev --with-yajl --with-sanlock --with-libpcap
>     > --with-macvtap --with-audit --with-dtrace --with-driver-modules
>     > --with-firewalld --with-firewalld-zone --without-wireshark-dissector
>     > --without-pm-utils --with-nss-plugin '--with-packager=Unknown,
>     > 2019-08-19-12:13:01, lab.rhel8.me <http://lab.rhel8.me>
>     <http://lab.rhel8.me>'
>     > --with-packager-version=1.el8 --with-qemu-user=qemu
>     > --with-qemu-group=qemu --with-tls-priority=@LIBVIRT,SYSTEM
>     > --enable-werror --enable-expensive-tests --with-init-script=systemd
>     > --without-login-shell && make
>     >
>     > Start libvirtd and virtlogd
>     > # LD_PRELOAD="$(find src -name '*.so.*'|tr '\n' ' ')"
>     src/.libs/libvirtd
>     > # LD_PRELOAD="$(find src -name '*.so.*'|tr '\n' ' ')" ./src/virtlogd
>     >
>     > Then try to list all domains:
>     > # virsh list --all
>     >
>     > Libvirtd exits with segment fault:
>     > [1]    30104 segmentation fault (core dumped)  LD_PRELOAD="$(find src
>     > -name '*.so.*'|tr '\n' ' ')" src/.libs/libvirtd
>     >
>     > Version:
>     > qemu-4.1
>     >
>     > Backtrace:
>     > (gdb) bt
>     > #0  0x00007fbe57a0d1b9 in virDomainVirtioSerialAddrSetAddControllers
>     > (def=<optimized out>, def=<optimized out>, addrs=<optimized out>) at
>     > conf/domain_addr.c:1656
>     > #1  virDomainVirtioSerialAddrSetCreateFromDomain
>     > (def=def@entry=0x7fbde81cc3f0) at conf/domain_addr.c:1753
>     > #2  0x00007fbe0179897e in qemuDomainAssignVirtioSerialAddresses
>     > (def=0x7fbde81cc3f0) at qemu/qemu_domain_address.c:3174
>     > #3  qemuDomainAssignAddresses (def=0x7fbde81cc3f0,
>     > qemuCaps=0x7fbde81d2210, driver=0x7fbde8126850, obj=0x0,
>     > newDomain=<optimized out>) at qemu/qemu_domain_address.c:3174
>     > #4  0x00007fbe57a39e0d in virDomainDefPostParse
>     > (def=def@entry=0x7fbde81cc3f0, caps=caps@entry=0x7fbde8154d20,
>     > parseFlags=parseFlags@entry=4610, xmlopt=xmlopt@entry=0x7fbde83ce070,
>     >      parseOpaque=parseOpaque@entry=0x0) at conf/domain_conf.c:5858
>     > #5  0x00007fbe57a525c5 in virDomainDefParseNode (xml=<optimized out>,
>     > root=0x7fbde83c5ff0, caps=0x7fbde8154d20, xmlopt=0x7fbde83ce070,
>     > parseOpaque=0x0, flags=4610) at conf/domain_conf.c:21677
>     > #6  0x00007fbe57a526c8 in virDomainDefParse (xmlStr=xmlStr@entry=0x0,
>     > filename=<optimized out>, caps=caps@entry=0x7fbde8154d20,
>     > xmlopt=xmlopt@entry=0x7fbde83ce070, parseOpaque=parseOpaque@entry=0x0,
>     >      flags=flags@entry=4610) at conf/domain_conf.c:21628
>     > #7  0x00007fbe57a528f6 in virDomainDefParseFile (filename=<optimized
>     > out>, caps=caps@entry=0x7fbde8154d20,
>     > xmlopt=xmlopt@entry=0x7fbde83ce070,
>     parseOpaque=parseOpaque@entry=0x0,
>     > flags=flags@entry=4610)
>     >      at conf/domain_conf.c:21653
>     > #8  0x00007fbe57a5e16a in virDomainObjListLoadConfig (opaque=0x0,
>     > notify=0x0, name=0x7fbde81d7ff3 "pc", autostartDir=0x7fbde8124070
>     > "/etc/libvirt/qemu/autostart", configDir=0x7fbde8124050
>     > "/etc/libvirt/qemu",
>     >      xmlopt=0x7fbde83ce070, caps=0x7fbde8154d20,
>     doms=0x7fbde8126940) at
>     > conf/virdomainobjlist.c:503
>     > #9  virDomainObjListLoadAllConfigs (doms=0x7fbde8126940,
>     > configDir=0x7fbde8124050 "/etc/libvirt/qemu",
>     > autostartDir=0x7fbde8124070 "/etc/libvirt/qemu/autostart",
>     > liveStatus=liveStatus@entry=false,
>     >      caps=0x7fbde8154d20, xmlopt=0x7fbde83ce070, notify=0x0,
>     opaque=0x0)
>     > at conf/virdomainobjlist.c:625
>     > #10 0x00007fbe017f57e2 in qemuStateInitialize (privileged=true,
>     > callback=<optimized out>, opaque=<optimized out>) at
>     > qemu/qemu_driver.c:1007
>     > #11 0x00007fbe57b8033d in virStateInitialize (privileged=true,
>     > mandatory=mandatory@entry=false,
>     callback=callback@entry=0x55dfb702ecc0
>     > <daemonInhibitCallback>, opaque=opaque@entry=0x55dfb8869d60)
>     >      at libvirt.c:666
>     > #12 0x000055dfb702ed1d in daemonRunStateInit
>     (opaque=0x55dfb8869d60) at
>     > remote/remote_daemon.c:846
>     > #13 0x00007fbe579f4be2 in virThreadHelper (data=<optimized out>) at
>     > util/virthread.c:196
>     > #14 0x00007fbe55a322de in start_thread (arg=<optimized out>) at
>     > pthread_create.c:486
>     > #15 0x00007fbe55763133 in clone () at
>     > ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>     >
>     > Could you please check this issue?
>     > The full threads backtrace is in attachment
>     >
>
> Hello, the git bisect shows that is the first bad commit:
> 192229f3a76ccc1b98a2c9e24f1feb0465b87a0b is the first bad commit
> commit 192229f3a76ccc1b98a2c9e24f1feb0465b87a0b
> Author: Cole Robinson <crobinso@redhat.com <mailto:crobinso@redhat.com>>
> Date:   Fri Oct 4 19:57:55 2019 -0400
>
>     storagefile: Push extension_end calc to qcow2GetBackingStoreFormat
>
>     This is a step towards making this qcow2GetBackingStoreFormat into
>     a generic qcow2 extensions parser
>
>     Signed-off-by: Cole Robinson <crobinso@redhat.com
> <mailto:crobinso@redhat.com>>
>
>
> Steps:
> 1. Merge crobinso/qcow2-data_file branch to 37b565c00.
> 2. Copy .gdbinit to libvirt source dir. Change the arguments values of 
> check-segv.sh
> 3. Set v5.8.0 as the start of bisect. Then start bisect.
> # git bisect start HEAD v5.8.0
> # git bisect run /tmp/check-segv.sh
>

I'm still quite confused. Maybe something I'm missing in one of these
commits is causing memory corruption that is manifesting elsewhere?

Can you provide full LIBVIRT_DEBUG=1 output when starting libvirtd? You
can use git master now because the patches have been pushed. I suggest
hosting the output somewhere rather than attaching it here, because it
will probably be large

Also, if you can pinpoint what VM XML that is being parsed when this
crashes, and post that too, it might help.

Thanks,
Cole