Re: [libvirt] [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg <danken@redhat.com> wrote:
On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg <danken@redhat.com> wrote:
On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:
Hi Dan,
Certainly, I've uploaded them to fedora's paste bin and tried to snip just the relevant details.
Sender (hv01.melb.domain.net): http://paste.fedoraproject.org/39660/92339651/
This one has
libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tls://hv02.melb.domain.net/system
which is most often related to firewall issues, and some time to key mismatch.
Does virsh -c qemu+tls://hv02.melb.domain.net/system capabilities work when run from the command line of hv01?
Dan.
Receiver (hv02.melb.domain.net): ` http://paste.fedoraproject.org/39661/23406913/
VM being transfered is ovirt_guest_vm
Thanks, Andrew
virsh -c qemu+tls://hv02.melb.domain.net/system 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: 0.10.2, package: 18.el6_4.9 (CentOS BuildSystem <http://bugs.centos.org>, 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org) 2013-09-15 10:41:10.620+0000: 23994: warning : virNetTLSContextCheckCertificate:1102 : Certificate check failed Certificate failed validation: The certificate hasn't got a known issuer.
Would you share your
openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -text
openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
on both hosts? This content may be sensitive, and may not provide an answer why libvirt on src cannot contact libvirtd on the other host. So before you do that, would you test if
vdsClient -s hv02.melb.domain.net getVdsCapabilities
works when run on hv01? It may be that the certificates are fine, but libvirt is not configured to use the correct ones.
Dan.
vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
I did a quick comparison between the files on both hosts, they seem to have the right details (host names, authority etc.) cacert.pem matches
/etc/libvirt/libvirtd.conf
ca_file="/etc/pki/vdsm/certs/cacert.pem" cert_file="/etc/pki/vdsm/certs/vdsmcert.pem" key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
Maybe someone on libvir-list could guess why this could be happening?
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Andrew Lau" <andrew@andrewklau.com> Cc: libvir-list@redhat.com, "users" <users@ovirt.org> Sent: Sunday, September 15, 2013 3:47:03 PM Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg <danken@redhat.com> wrote:
On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg <danken@redhat.com> wrote:
On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:
Hi Dan,
Certainly, I've uploaded them to fedora's paste bin and tried to snip just the relevant details.
Sender (hv01.melb.domain.net): http://paste.fedoraproject.org/39660/92339651/
This one has
libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tls://hv02.melb.domain.net/system
which is most often related to firewall issues, and some time to key mismatch.
Does virsh -c qemu+tls://hv02.melb.domain.net/system capabilities work when run from the command line of hv01?
Dan.
Receiver (hv02.melb.domain.net): ` http://paste.fedoraproject.org/39661/23406913/
VM being transfered is ovirt_guest_vm
Thanks, Andrew
virsh -c qemu+tls://hv02.melb.domain.net/system 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: 0.10.2, package: 18.el6_4.9 (CentOS BuildSystem <http://bugs.centos.org>, 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org) 2013-09-15 10:41:10.620+0000: 23994: warning : virNetTLSContextCheckCertificate:1102 : Certificate check failed Certificate failed validation: The certificate hasn't got a known issuer.
Would you share your
openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -text
openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
on both hosts? This content may be sensitive, and may not provide an answer why libvirt on src cannot contact libvirtd on the other host. So before you do that, would you test if
vdsClient -s hv02.melb.domain.net getVdsCapabilities
works when run on hv01? It may be that the certificates are fine, but libvirt is not configured to use the correct ones.
Dan.
vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
I did a quick comparison between the files on both hosts, they seem to have the right details (host names, authority etc.) cacert.pem matches
/etc/libvirt/libvirtd.conf
ca_file="/etc/pki/vdsm/certs/cacert.pem" cert_file="/etc/pki/vdsm/certs/vdsmcert.pem" key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
this sounds a little like https://bugzilla.redhat.com/show_bug.cgi?id=996146 can you try to restart libvirt (on both hosts just to be sure) and try again?
Maybe someone on libvir-list could guess why this could be happening? _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel <ofrenkel@redhat.com> wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Andrew Lau" <andrew@andrewklau.com> Cc: libvir-list@redhat.com, "users" <users@ovirt.org> Sent: Sunday, September 15, 2013 3:47:03 PM Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg <danken@redhat.com> wrote:
On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg <danken@redhat.com
wrote:
On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote: > Hi Dan, > > Certainly, I've uploaded them to fedora's paste bin and tried
to
> snip just > the relevant details. > > Sender (hv01.melb.domain.net): > http://paste.fedoraproject.org/39660/92339651/
This one has
libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tls://hv02.melb.domain.net/system
which is most often related to firewall issues, and some time to key mismatch.
Does virsh -c qemu+tls://hv02.melb.domain.net/system capabilities work when run from the command line of hv01?
Dan. > Receiver (hv02.melb.domain.net): ` > http://paste.fedoraproject.org/39661/23406913/ > > VM being transfered is ovirt_guest_vm > > Thanks, > Andrew
virsh -c qemu+tls://hv02.melb.domain.net/system 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: 0.10.2, package: 18.el6_4.9 (CentOS BuildSystem <http://bugs.centos.org>, 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org) 2013-09-15 10:41:10.620+0000: 23994: warning : virNetTLSContextCheckCertificate:1102 : Certificate check failed Certificate failed validation: The certificate hasn't got a known issuer.
Would you share your
openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -text
openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
on both hosts? This content may be sensitive, and may not provide an answer why libvirt on src cannot contact libvirtd on the other host. So before you do that, would you test if
vdsClient -s hv02.melb.domain.net getVdsCapabilities
works when run on hv01? It may be that the certificates are fine, but libvirt is not configured to use the correct ones.
Dan.
vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
I did a quick comparison between the files on both hosts, they seem to have the right details (host names, authority etc.) cacert.pem matches
/etc/libvirt/libvirtd.conf
ca_file="/etc/pki/vdsm/certs/cacert.pem" cert_file="/etc/pki/vdsm/certs/vdsmcert.pem" key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
this sounds a little like https://bugzilla.redhat.com/show_bug.cgi?id=996146
can you try to restart libvirt (on both hosts just to be sure) and try again?
Maybe someone on libvir-list could guess why this could be happening? _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I did try that already service vdsmd restart [root@hv02 ~]# service vdsmd restart Shutting down vdsm daemon: vdsm watchdog stop [ OK ] vdsm stop [ OK ] Starting configure libvirt to VDSM ... libvirt is already configured for vdsm =Done configuring libvirt= libvir: Network Filter Driver error : Requested operation is not valid: nwfilter is in use Checking conflicts ... SUCCESS: ssl configured to true. No conflicts Starting up vdsm daemon: vdsm start [ OK ] Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes migration was working fine. Only when I upgraded to the nightly and it picked up the new vdsm packages it started to fail.
----- Original Message -----
From: "Andrew Lau" <andrew@andrewklau.com> To: "Omer Frenkel" <ofrenkel@redhat.com> Cc: "Dan Kenigsberg" <danken@redhat.com>, libvir-list@redhat.com, "users" <users@ovirt.org> Sent: Monday, September 16, 2013 1:38:53 AM Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel < ofrenkel@redhat.com > wrote:
----- Original Message -----
From: "Dan Kenigsberg" < danken@redhat.com >
To: "Andrew Lau" < andrew@andrewklau.com >
Cc: libvir-list@redhat.com , "users" < users@ovirt.org >
Sent: Sunday, September 15, 2013 3:47:03 PM
Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg < danken@redhat.com > wrote:
On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg < danken@redhat.com >
wrote:
> On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote:
> > Hi Dan,
> >
> > Certainly, I've uploaded them to fedora's paste bin and tried > > to
> > snip
> just
> > the relevant details.
> >
> > Sender ( hv01.melb.domain.net ):
>
> This one has
>
> libvirtError: operation failed: Failed to connect to remote
> libvirt
> URI qemu+tls:// hv02.melb.domain.net/system
>
> which is most often related to firewall issues, and some time to > key
> mismatch.
>
> Does
> virsh -c qemu+tls:// hv02.melb.domain.net/system capabilities
> work when run from the command line of hv01?
>
> Dan.
> > Receiver ( hv02.melb.domain.net ): `
> >
> > VM being transfered is ovirt_guest_vm
> >
> > Thanks,
> > Andrew
>
virsh -c qemu+tls:// hv02.melb.domain.net/system
2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: 0.10.2,
package: 18.el6_4.9 (CentOS BuildSystem < http://bugs.centos.org >,
2013-07-02-11:19:29, c6b8.bsys.dev.centos.org )
2013-09-15 10:41:10.620+0000: 23994: warning :
virNetTLSContextCheckCertificate:1102 : Certificate check failed
Certificate failed validation: The certificate hasn't got a known
issuer.
Would you share your
openssl x509 -in
/etc/pki/vdsm/certs/cacert.pem -text
openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
on both hosts? This content may be sensitive, and may not
provide an answer why libvirt on src cannot contact libvirtd on the
other host. So before you do that, would you test if
vdsClient -s hv02.melb.domain.net getVdsCapabilities
works when run on hv01? It may be that the certificates are fine, but
libvirt is not configured to use the correct ones.
Dan.
vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
I did a quick comparison between the files on both hosts, they seem to have
the right details (host names, authority etc.)
cacert.pem matches
/etc/libvirt/libvirtd.conf
ca_file="/etc/pki/vdsm/certs/cacert.pem"
cert_file="/etc/pki/vdsm/certs/vdsmcert.pem"
key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
this sounds a little like
can you try to restart libvirt (on both hosts just to be sure) and try again?
Maybe someone on libvir-list could guess why this could be happening?
_______________________________________________
Users mailing list
Users@ovirt.org
I did try that already
service vdsmd restart
[root@hv02 ~]# service vdsmd restart Shutting down vdsm daemon: vdsm watchdog stop [ OK ] vdsm stop [ OK ] Starting configure libvirt to VDSM ... libvirt is already configured for vdsm =Done configuring libvirt= libvir: Network Filter Driver error : Requested operation is not valid: nwfilter is in use Checking conflicts ... SUCCESS: ssl configured to true. No conflicts Starting up vdsm daemon: vdsm start [ OK ]
Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes migration was working fine. Only when I upgraded to the nightly and it picked up the new vdsm packages it started to fail.
can you try to restart the libvirtd service itself? not vdsm
On Mon, Sep 16, 2013 at 5:41 PM, Omer Frenkel <ofrenkel@redhat.com> wrote:
------------------------------
*From: *"Andrew Lau" <andrew@andrewklau.com> *To: *"Omer Frenkel" <ofrenkel@redhat.com> *Cc: *"Dan Kenigsberg" <danken@redhat.com>, libvir-list@redhat.com, "users" <users@ovirt.org> *Sent: *Monday, September 16, 2013 1:38:53 AM
*Subject: *Re: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 11:51 PM, Omer Frenkel <ofrenkel@redhat.com>wrote:
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Andrew Lau" <andrew@andrewklau.com> Cc: libvir-list@redhat.com, "users" <users@ovirt.org> Sent: Sunday, September 15, 2013 3:47:03 PM Subject: Re: [Users] Live Migration failed oVirt 3.3 Nightly
On Sun, Sep 15, 2013 at 09:57:47PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 9:34 PM, Dan Kenigsberg <danken@redhat.com> wrote:
On Sun, Sep 15, 2013 at 08:44:18PM +1000, Andrew Lau wrote:
On Sun, Sep 15, 2013 at 8:00 PM, Dan Kenigsberg < danken@redhat.com> wrote:
> On Sun, Sep 15, 2013 at 06:48:41PM +1000, Andrew Lau wrote: > > Hi Dan, > > > > Certainly, I've uploaded them to fedora's paste bin and tried to > > snip > just > > the relevant details. > > > > Sender (hv01.melb.domain.net): > > http://paste.fedoraproject.org/39660/92339651/ > > This one has > > libvirtError: operation failed: Failed to connect to remote > libvirt > URI qemu+tls://hv02.melb.domain.net/system > > which is most often related to firewall issues, and some time to key > mismatch. > > Does > virsh -c qemu+tls://hv02.melb.domain.net/systemcapabilities > work when run from the command line of hv01? > > Dan. > > Receiver (hv02.melb.domain.net): ` > > http://paste.fedoraproject.org/39661/23406913/ > > > > VM being transfered is ovirt_guest_vm > > > > Thanks, > > Andrew >
virsh -c qemu+tls://hv02.melb.domain.net/system 2013-09-15 10:41:10.620+0000: 23994: info : libvirt version: 0.10.2, package: 18.el6_4.9 (CentOS BuildSystem <http://bugs.centos.org>, 2013-07-02-11:19:29, c6b8.bsys.dev.centos.org) 2013-09-15 10:41:10.620+0000: 23994: warning : virNetTLSContextCheckCertificate:1102 : Certificate check failed Certificate failed validation: The certificate hasn't got a known issuer.
Would you share your
openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -text
openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text
on both hosts? This content may be sensitive, and may not provide an answer why libvirt on src cannot contact libvirtd on the other host. So before you do that, would you test if
vdsClient -s hv02.melb.domain.net getVdsCapabilities
works when run on hv01? It may be that the certificates are fine, but libvirt is not configured to use the correct ones.
Dan.
vdsClient -s hv02.melb.domain.net getVdsCapabilities runs fine
I did a quick comparison between the files on both hosts, they seem to have the right details (host names, authority etc.) cacert.pem matches
/etc/libvirt/libvirtd.conf
ca_file="/etc/pki/vdsm/certs/cacert.pem" cert_file="/etc/pki/vdsm/certs/vdsmcert.pem" key_file="/etc/pki/vdsm/keys/vdsmkey.pem"
this sounds a little like https://bugzilla.redhat.com/show_bug.cgi?id=996146
can you try to restart libvirt (on both hosts just to be sure) and try again?
Maybe someone on libvir-list could guess why this could be happening? _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I did try that already
service vdsmd restart
[root@hv02 ~]# service vdsmd restart Shutting down vdsm daemon: vdsm watchdog stop [ OK ] vdsm stop [ OK ] Starting configure libvirt to VDSM ... libvirt is already configured for vdsm =Done configuring libvirt= libvir: Network Filter Driver error : Requested operation is not valid: nwfilter is in use Checking conflicts ... SUCCESS: ssl configured to true. No conflicts Starting up vdsm daemon: vdsm start [ OK ]
Migration still failed. Keep in mind, when I had oVirt 3.3 on these nodes migration was working fine. Only when I upgraded to the nightly and it picked up the new vdsm packages it started to fail.
can you try to restart the libvirtd service itself? not vdsm
[root@hv01 ~]# service libvirtd restart Stopping libvirtd daemon: libvirtd: libvirtd is managed by upstart and started, use initctl instead [root@hv01 ~]# initctl restart libvirtd libvirtd start/running, process 4538 Migration was successful, thanks!
participants (3)
-
Andrew Lau -
Dan Kenigsberg -
Omer Frenkel