Re: [libvirt] [Users] Live Migration failed oVirt 3.3 Nightly

------------------------------ *From: *"Andrew Lau" <andrew(a)andrewklau.com&gt; *To: *"Omer Frenkel" <ofrenkel(a)redhat.com&gt; *Cc: *"Dan Kenigsberg" <danken(a)redhat.com&gt;, libvir-list(a)redhat.com, "users" <users(a)ovirt.org&gt; *Sent: *Monday, September 16, 2013 1:38:53 AM *Subject: *Re: [Users] Live Migration failed oVirt 3.3 Nightly On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel <ofrenkel(a)redhat.com&gt;wrote: > > > ----- Original Message ----- > > From: "Dan Kenigsberg" <danken(a)redhat.com&gt; > > To: "Andrew Lau" <andrew(a)andrewklau.com&gt; > > Cc: libvir-list(a)redhat.com, "users" <users(a)ovirt.org&gt; > > Sent: Sunday, September 15, 2013 3:47:03 PM > > Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly > > > > On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote: > > > On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg <danken(a)redhat.com&gt; > wrote: > > > > > > > On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote: > > > > > On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg < > danken(a)redhat.com&gt; > > > > wrote: > > > > > > > > > > > On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote: > > > > > > > Hi Dan, > > > > > > > > > > > > > > Certainly, I've uploaded them to fedora's paste bin and tried > to > > > > > > > snip > > > > > > just > > > > > > > the relevant details. > > > > > > > > > > > > > > Sender (hv01.melb.domain.net): > > > > > > > http://paste.fedoraproject.org/39660/92339651/ > > > > > > > > > > > > This one has > > > > > > > > > > > > libvirtError: operation failed: Failed to connect to remote > > > > > > libvirt > > > > > > URI qemu+tls://hv02.melb.domain.net/system > > > > > > > > > > > > which is most often related to firewall issues, and some time > to key > > > > > > mismatch. > > > > > > > > > > > > Does > > > > > > virsh -c qemu+tls://hv02.melb.domain.net/systemcapabilities > > > > > > work when run from the command line of hv01? > > > > > > > > > > > > Dan. > > > > > > > Receiver (hv02.melb.domain.net): ` > > > > > > > http://paste.fedoraproject.org/39661/23406913/ > > > > > > > > > > > > > > VM being transfered is ovirt_guest_vm > > > > > > > > > > > > > > Thanks, > > > > > > > Andrew > > > > > > > > > > > > > > > > virsh -c qemu+tls://hv02.melb.domain.net/system > > > > > 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: > 0.10.2, > > > > > package: 18.el6_4.9 (CentOS BuildSystem <http://bugs.centos.org>;, > > > > > 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org) > > > > > 2013-09-15 10:41:10.620+0000: 23994: warning : > > > > > virNetTLSContextCheckCertificate:1102 : Certificate check failed > > > > > Certificate failed validation: The certificate hasn't got a known > > > > > issuer. > > > > > > > > Would you share your > > > > > > > > > > > > openssl x509 -in > > > > /etc/pki/vdsm/certs/cacert.pem -text > > > > > > > > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text > > > > > > > > on both hosts? This content may be sensitive, and may not > > > > provide an answer why libvirt on src cannot contact libvirtd on the > > > > other host. So before you do that, would you test if > > > > > > > > > > > > vdsClient -s hv02.melb.domain.net getVdsCapabilities > > > > > > > > works when run on hv01? It may be that the certificates are fine, > but > > > > libvirt is not configured to use the correct ones. > > > > > > > > Dan. > > > > > > > > > > > vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine > > > > > > I did a quick comparison between the files on both hosts, they seem > to have > > > the right details (host names, authority etc.) > > > cacert.pem matches > > > > > > /etc/libvirt/libvirtd.conf > > > > > > ca_file="/etc/pki/vdsm/certs/cacert.pem" > > > cert_file="/etc/pki/vdsm/certs/vdsmcert.pem" > > > key_file="/etc/pki/vdsm/keys/vdsmkey.pem" > > > > this sounds a little like > https://bugzilla.redhat.com/show_bug.cgi?id=996146 > > can you try to restart libvirt (on both hosts just to be sure) and try > again? > > > Maybe someone on libvir-list could guess why this could be happening? > > _______________________________________________ > > Users mailing list > > Users(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > > > I did try that already service vdsmd restart [root@hv02 ~]# service vdsmd restart Shutting down vdsm daemon: vdsm watchdog stop [ OK ] vdsm stop [ OK ] Starting configure libvirt to VDSM ... libvirt is already configured for vdsm =Done configuring libvirt= libvir: Network Filter Driver error : Requested operation is not valid: nwfilter is in use Checking conflicts ... SUCCESS: ssl configured to true. No conflicts Starting up vdsm daemon: vdsm start [ OK ] Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes migration was working fine. Only when I upgraded to the nightly and it picked up the new vdsm packages it started to fail. can you try to restart the libvirtd service itself? not vdsm