Re: [libvirt] [Users] Live Migration failed oVirt 3.3 Nightly

From: "Andrew Lau" <andrew(a)andrewklau.com&gt; To: "Omer Frenkel" <ofrenkel(a)redhat.com&gt; Cc: "Dan Kenigsberg" <danken(a)redhat.com&gt;, libvir-list(a)redhat.com, "users" <users(a)ovirt.org&gt; Sent: Monday, September 16, 2013 1:38:53 AM Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly

On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel < ofrenkel(a)redhat.com > wrote:

> ----- Original Message ----- > > From: "Dan Kenigsberg" < danken(a)redhat.com > > > To: "Andrew Lau" < andrew(a)andrewklau.com > > > Cc: libvir-list(a)redhat.com , "users" < users(a)ovirt.org > > > Sent: Sunday, September 15, 2013 3:47:03 PM > > Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly > > > > On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote: > > > On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg < danken(a)redhat.com > > > > wrote: > > > > > > > On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote: > > > > > On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg < danken(a)redhat.com > > > > > > > > > > wrote: > > > > > > > > > > > On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote: > > > > > > > Hi Dan, > > > > > > > > > > > > > > Certainly, I've uploaded them to fedora's paste bin and tried > > > > > > > to > > > > > > > snip > > > > > > just > > > > > > > the relevant details. > > > > > > > > > > > > > > Sender ( hv01.melb.domain.net ): > > > > > > > http://paste.fedoraproject.org/39660/92339651/ > > > > > > > > > > > > This one has > > > > > > > > > > > > libvirtError: operation failed: Failed to connect to remote > > > > > > libvirt > > > > > > URI qemu+tls:// hv02.melb.domain.net/system > > > > > > > > > > > > which is most often related to firewall issues, and some time to > > > > > > key > > > > > > mismatch. > > > > > > > > > > > > Does > > > > > > virsh -c qemu+tls:// hv02.melb.domain.net/system capabilities > > > > > > work when run from the command line of hv01? > > > > > > > > > > > > Dan. > > > > > > > Receiver ( hv02.melb.domain.net ): ` > > > > > > > http://paste.fedoraproject.org/39661/23406913/ > > > > > > > > > > > > > > VM being transfered is ovirt_guest_vm > > > > > > > > > > > > > > Thanks, > > > > > > > Andrew > > > > > > > > > > > > > > > > virsh -c qemu+tls:// hv02.melb.domain.net/system > > > > > 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: > > > > > 0.10.2, > > > > > package: 18.el6_4.9 (CentOS BuildSystem < http://bugs.centos.org >, > > > > > 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org ) > > > > > 2013-09-15 10:41:10.620+0000: 23994: warning : > > > > > virNetTLSContextCheckCertificate:1102 : Certificate check failed > > > > > Certificate failed validation: The certificate hasn't got a known > > > > > issuer. > > > > > > > > Would you share your > > > > > > > > > > > > openssl x509 -in > > > > /etc/pki/vdsm/certs/cacert.pem -text > > > > > > > > openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text > > > > > > > > on both hosts? This content may be sensitive, and may not > > > > provide an answer why libvirt on src cannot contact libvirtd on the > > > > other host. So before you do that, would you test if > > > > > > > > > > > > vdsClient -s hv02.melb.domain.net getVdsCapabilities > > > > > > > > works when run on hv01? It may be that the certificates are fine, but > > > > libvirt is not configured to use the correct ones. > > > > > > > > Dan. > > > > > > > > > > > vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine > > > > > > I did a quick comparison between the files on both hosts, they seem to > > > have > > > the right details (host names, authority etc.) > > > cacert.pem matches > > > > > > /etc/libvirt/libvirtd.conf > > > > > > ca_file="/etc/pki/vdsm/certs/cacert.pem" > > > cert_file="/etc/pki/vdsm/certs/vdsmcert.pem" > > > key_file="/etc/pki/vdsm/keys/vdsmkey.pem" > >

> this sounds a little like > https://bugzilla.redhat.com/show_bug.cgi?id=996146

> can you try to restart libvirt (on both hosts just to be sure) and try > again?

> > Maybe someone on libvir-list could guess why this could be happening? > > _______________________________________________ > > Users mailing list > > Users(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > >

I did try that already

service vdsmd restart

[root@hv02 ~]# service vdsmd restart Shutting down vdsm daemon: vdsm watchdog stop [ OK ] vdsm stop [ OK ] Starting configure libvirt to VDSM ... libvirt is already configured for vdsm =Done configuring libvirt= libvir: Network Filter Driver error : Requested operation is not valid: nwfilter is in use Checking conflicts ... SUCCESS: ssl configured to true. No conflicts Starting up vdsm daemon: vdsm start [ OK ]

Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes migration was working fine. Only when I upgraded to the nightly and it picked up the new vdsm packages it started to fail.