Daniel P. Berrange wrote:

...

But when accessing remotely, I get no useful error, and a hang:

$ virsh -c qemu+ssh://jim@server/system libvir: Remote error : authentication failed <process hangs here>

$ virsh --readonly -c qemu+ssh://jim@server/system libvir: Remote error : authentication failed <process hangs here>

Furthermore, on the server, this leaves "nc" processes running, and eventually there are enough that libvirtd stops accepting new connections.

The hang is really odd. That suggests something is not closing the socket connection properly. If you had been yusing 0.6.1/.2/.3 I would have said it was one of the libvirtd bugs, but 0.6.4 has all event handling bugs fixed. Perhaps the libvirtd client is not killing the SSH session / process when it closes the connection after auth failure.

I was using 0.4.6 on the client side. I upgraded that to 0.6.4, but I still get the hang. Virsh prints nothing; the LIBVIRT_DEBUG output is: 17:34:58.524: debug : doRemoteOpen:505 : proceeding with name = qemu:///system 17:34:58.525: debug : virExecWithHook:573 : ssh server nc -U /var/run/libvirt/libvirt-sock 17:34:58.526: debug : call:6947 : Doing call 66 (nil) 17:34:58.527: debug : call:7017 : We have the buck 66 0x7fba56729010 0x7fba56729010 17:34:59.359: debug : processCallRecvLen:6605 : Got length, now need 36 total (32 more) 17:34:59.360: debug : processCalls:6873 : Giving up the buck 66 0x7fba56729010 (nil) 17:34:59.360: debug : call:7048 : All done with our call 66 (nil) 0x7fba56729010 17:34:59.360: debug : remoteAuthPolkit:6114 : Client initialize PolicyKit authentication 17:34:59.360: debug : call:6947 : Doing call 70 (nil) 17:34:59.360: debug : call:7017 : We have the buck 70 0xbccef0 0xbccef0 17:34:59.433: debug : processCallRecvLen:6605 : Got length, now need 128 total (124 more) 17:34:59.434: debug : processCalls:6873 : Giving up the buck 70 0xbccef0 (nil) 17:34:59.434: debug : call:7048 : All done with our call 70 (nil) 0xbccef0 17:34:59.434: error : server_error:7231 : authentication failed 17:35:13.585: debug : do_open:999 : driver 4 remote returned ERROR 17:35:13.585: debug : virUnrefConnect:232 : unref connection 0xbc6a60 1 17:35:13.585: debug : virReleaseConnect:191 : release connection 0xbc6a60 If I kill the libvirtd process on the server, the client then finally prints: error: authentication failed error: failed to connect to the hypervisor and the client then exits. On the server side, the libvirtd output is 17:34:59.378: debug : remoteDispatchAuthPolkit:3385 : Start PolicyKit auth 25 17:34:59.378: info : remoteDispatchAuthPolkit:3396 : Checking PID 7551 running as 1000 17:34:59.379: debug : virEventRunOnce:567 : Poll got 1 event 17:34:59.379: debug : virEventDispatchHandles:450 : Dispatch n=2 f=9 w=3 e=1 0x1a72790 17:34:59.379: debug : nodeDeviceLock:52 : LOCK node 0x1a748e0 17:34:59.379: debug : nodeDeviceUnlock:57 : UNLOCK node 0x1a748e0 17:34:59.426: error : remoteDispatchAuthPolkit:3451 : Policy kit denied action org.libvirt.unix.manage from pid 7551, uid 1000, result: auth_admin_keep_session The hang aside, it seems libvirtd should be using org.libvirt.unix.monitor for the readonly connection?

...

Is policykit authentication supposed to work over qemu+ssh?

Yes, but only if you ssh as root such that policykit is a no-op.

The problem you are seeing is becaue you SSH as non-root. PolicyKit relies on ConsoleKit to determine who is authorized, and SSH does not register ConsoleKit Sessions.

As I mentioned, I've modified the PolicyKit libvirtd configuration to not require a session: <match action="org.libvirt.unix.manage"> <return result="auth_admin_keep_session"/> </match> so I was hoping that wouldn't be a problem. With this configuration, I think even using libpam-ck-connector wouldn't change things?

...

I was hoping it would at least not break the --readonly case.

That all said --readonly is intended to work at all times. Our default policy file includes a rule <allow_any>yes</allow_any> which is telling policykit to allow access even if the client is not associatied with any ConsoleKit session. So this should have allowed it to work for you with --readonly.

Right, it seems libvirtd is missing readonly somehow? -jim

On Wed, Jun 17, 2009 at 07:27:22PM -0400, Jim Paris wrote:

I wrote:

...

...

Ok, this bit definitely sounds like a server side bug, unless perhaps there is some buffering taking place in ssh or nc causing the errore reply packet to not be send back promptly

I'll try to get some better traces of what's going on here.

The error is getting back to the client. On the client, remoteAuthenticate does fail and return -1. The client then ends up blocked in the waitpid at remote_internal.c:877:

865 failed: 866 /* Close the socket if we failed. */ 867 if (priv->sock >= 0) { 868 if (priv->uses_tls && priv->session) { 869 gnutls_bye (priv->session, GNUTLS_SHUT_RDWR); 870 gnutls_deinit (priv->session); 871 } 872 close (priv->sock); 873 #ifndef WIN32 874 if (priv->pid > 0) { 875 pid_t reap; 876 do { 877 reap = waitpid(priv->pid, NULL, 0); 878 if (reap == -1 && errno == EINTR) 879 continue; 880 } while (reap != -1 && reap != priv->pid); 881 } 882 #endif 883 }

Nothing gets printed up until this point, which is why there's no output.

I guess the client is waiting for SSH to die, which isn't happening for some reason. That must be a bug on the server side, although the client should also probably be more robust in this case..

We close the socket to the 'nc' process here so in theory it should be getting a HUP event from poll or EOF from read, etc and then exiting. Ominously though I see several patches to Fedora's 'nc' RPM at least one of which is related to nc hanging forever after getting HUP fback from poll(). What distro are you using ? http://cvs.fedoraproject.org/viewvc/rpms/nc/F-11/ Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On Thu, Jun 18, 2009 at 12:20:40PM -0400, Jim Paris wrote:

Daniel P. Berrange wrote:

...

We close the socket to the 'nc' process here so in theory it should be getting a HUP event from poll or EOF from read, etc and then exiting. Ominously though I see several patches to Fedora's 'nc' RPM at least one of which is related to nc hanging forever after getting HUP fback from poll(). What distro are you using ?

http://cvs.fedoraproject.org/viewvc/rpms/nc/F-11/

I'm using Debian. I've already had to switch from the "netcat-traditional" package to the "netcat-openbsd" package. Debian does already include that patch, but what a mess...

I know the reason why it gets stuck on the server end too - after an auth failure, the server won't kick off the client. The connection just remains in an unauthenticated state. This allows the client to (in theory) retry the authentication step, and gives us a little more flexibility for any future protocol changes we might need to make. I think the best way to solve the problem of 'nc' potentially not quitting promptly, is to simply have the remote client kill() the SSH client pid, rather than simply closing the socket & doing waitpid() on the SSH client. This would ensure the waitpid promptly cleans up.

Since already know libvirtd is installed on the remote host, would it make sense to just add a new set of options: libvirtd --socket-connect libvirtd --socket-connect-ro that do the same thing as "nc -U" on the appropriate socket? Then we know it would work everywhere, and have the added benefit that the client wouldn't need to know the location of the socket.

If we'd thought of this originally, I would certainly have done it this way, but if we did this now, it would break compatability. ie new libvirt clients would be trying to run a binary that does not exist with old server deployments. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On Thu, Jun 18, 2009 at 12:20:40PM -0400, Jim Paris wrote:

Daniel P. Berrange wrote:

...

We close the socket to the 'nc' process here so in theory it should be getting a HUP event from poll or EOF from read, etc and then exiting. Ominously though I see several patches to Fedora's 'nc' RPM at least one of which is related to nc hanging forever after getting HUP fback from poll(). What distro are you using ?

http://cvs.fedoraproject.org/viewvc/rpms/nc/F-11/

I'm using Debian. I've already had to switch from the "netcat-traditional" package to the "netcat-openbsd" package. Debian does already include that patch, but what a mess...

Since already know libvirtd is installed on the remote host, would it make sense to just add a new set of options: libvirtd --socket-connect libvirtd --socket-connect-ro that do the same thing as "nc -U" on the appropriate socket? Then we know it would work everywhere, and have the added benefit that the client wouldn't need to know the location of the socket. Last time I checked Debian's nc wouldn't properly close the scoket on EOF so I added this to virt-manager:

- argv += [ server, "nc", vncaddr, str(vncport) ] + argv += [ server, "nc", "-q", "0", vncaddr, str(vncport) ] The problem you're seeing suggests that we need this in virsh too. Could you check if this helps? Cheers, -- Guido