Can RHEL7 VM run remote commands to Fedora36 host?
I have a Fedora36 laptop which hosts VMs with RHEL7 using libvirt. One of the RHEL7 VMs, runs remote commands (as root) to 'start' another VM by way of my laptop. In other words, the following command is run: virsh --connect 'qemu+ssh://192.168.120.1/system' start beaker-test-vm1.beaker If I run non-remote version of the command on the laptop, it is successful. For example, virsh --connect qemu:///system start beaker-test-vm1.beaker <-- Successful on laptop. If I do a query like the following *(notice socket use)*, it is successful. virsh -d0 --connect 'qemu+ssh://192.168.120.1/system?*socket*=/var/run/libvirt/libvirt-sock-ro' domstate beaker-test-vm1.beaker Without socket, I get the following error: *error: failed to connect to the hypervisor* *error: End of file while reading data: Ncat: No such file or directory.: Input/output error* This does not work for 'start' because I believe this is a read-only socket since I see the error: error: Failed to start domain beaker-test-vm1.beaker error: operation forbidden: read only access prevents virDomainCreate When I look at my laptop, there is no /var/run/libvirt/libvirt-sock. So.....I've been wondering whether RHEL7 virsh/libvirt is compatible with Fedora36. Is there a work-around? I can't change the distros on my laptop or VMs. Carol
Well I don't know if this helps, but it's there a reason you haven't tried running the command that succeeds on the laptop directly over SSH? EG: ssh LAPTOPUSER@Laptop "virsh --connect qemu:///system start beaker-test-vm1.beaker" Ben S On Thu, Jul 28, 2022, 11:16 AM Carol Bouchard <cbouchar@redhat.com> wrote:
I have a Fedora36 laptop which hosts VMs with RHEL7 using libvirt. One of the RHEL7 VMs, runs remote commands (as root) to 'start' another VM by way of my laptop. In other words, the following command is run: virsh --connect 'qemu+ssh://192.168.120.1/system' start beaker-test-vm1.beaker If I run non-remote version of the command on the laptop, it is successful. For example, virsh --connect qemu:///system start beaker-test-vm1.beaker <-- Successful on laptop.
If I do a query like the following *(notice socket use)*, it is successful. virsh -d0 --connect 'qemu+ssh://192.168.120.1/system?*socket*=/var/run/libvirt/libvirt-sock-ro' domstate beaker-test-vm1.beaker
Without socket, I get the following error:
*error: failed to connect to the hypervisor*
*error: End of file while reading data: Ncat: No such file or directory.: Input/output error*
This does not work for 'start' because I believe this is a read-only socket since I see the error: error: Failed to start domain beaker-test-vm1.beaker error: operation forbidden: read only access prevents virDomainCreate
When I look at my laptop, there is no /var/run/libvirt/libvirt-sock. So.....I've been wondering whether RHEL7 virsh/libvirt is compatible with Fedora36. Is there a work-around? I can't change the distros on my laptop or VMs.
Carol
On Wed, Jul 27, 2022 at 01:18:00PM -0400, Carol Bouchard wrote:
I have a Fedora36 laptop which hosts VMs with RHEL7 using libvirt. One of the RHEL7 VMs, runs remote commands (as root) to 'start' another VM by way of my laptop. In other words, the following command is run: virsh --connect 'qemu+ssh://192.168.120.1/system' start beaker-test-vm1.beaker If I run non-remote version of the command on the laptop, it is successful. For example, virsh --connect qemu:///system start beaker-test-vm1.beaker <-- Successful on laptop.
If I do a query like the following *(notice socket use)*, it is successful. virsh -d0 --connect 'qemu+ssh://192.168.120.1/system?*socket*=/var/run/libvirt/libvirt-sock-ro' domstate beaker-test-vm1.beaker
Without socket, I get the following error:
*error: failed to connect to the hypervisor*
*error: End of file while reading data: Ncat: No such file or directory.: Input/output error*
This does not work for 'start' because I believe this is a read-only socket since I see the error: error: Failed to start domain beaker-test-vm1.beaker error: operation forbidden: read only access prevents virDomainCreate
When I look at my laptop, there is no /var/run/libvirt/libvirt-sock. So.....I've been wondering whether RHEL7 virsh/libvirt is compatible with Fedora36. Is there a work-around? I can't change the distros on my laptop or VMs.
Hello, since Fedora 35 libvirt has used the modular daemons architecture. What this means for you (and appears very confusing) is that when you try starting a VM locally on your F36 laptop, your virsh client doesn't connect to libvirt-sock anymore, there's a dedicated connection socket for each of the daemons now and instead will connect to virtqemud-sock. Now, old virsh clients like the one you have on your RHEL7 don't know about this and expect to connect to libvirt-sock instead. In order to create that socket and restore functionality for old clients you need to start and enable the virtproxyd.socket systemd unit which proxies old client connections to the new sockets we have. Why the virtproxyd socket isn't running by default unless you disabled it beats me, since: $ systemctl status virtproxyd Loaded: loaded (/usr/lib/systemd/system/virtproxyd.socket; enabled; vendor preset: enabled) ^^^here^^^ is set correctly after installation. Anyhow, just do: $ sudo systemctl enable --now virtproxyd.socket on your laptop and you're good to go Regards, Erik So, if you don't have the libvirt-sock created that means the virtproxyd.socket systemd unit isn't active. Just enable the socket and try again. Here, I simulated it for you with my VMs: VM1: $ cat /etc/os-release NAME="Fedora Linux" VERSION="36 (Thirty Six)" ... $
Carol
On Fri, Jul 29, 2022 at 08:18:17AM +0200, Erik Skultety wrote:
On Wed, Jul 27, 2022 at 01:18:00PM -0400, Carol Bouchard wrote:
I have a Fedora36 laptop which hosts VMs with RHEL7 using libvirt. One of the RHEL7 VMs, runs remote commands (as root) to 'start' another VM by way of my laptop. In other words, the following command is run: virsh --connect 'qemu+ssh://192.168.120.1/system' start beaker-test-vm1.beaker If I run non-remote version of the command on the laptop, it is successful. For example, virsh --connect qemu:///system start beaker-test-vm1.beaker <-- Successful on laptop.
If I do a query like the following *(notice socket use)*, it is successful. virsh -d0 --connect 'qemu+ssh://192.168.120.1/system?*socket*=/var/run/libvirt/libvirt-sock-ro' domstate beaker-test-vm1.beaker
Without socket, I get the following error:
*error: failed to connect to the hypervisor*
*error: End of file while reading data: Ncat: No such file or directory.: Input/output error*
This does not work for 'start' because I believe this is a read-only socket since I see the error: error: Failed to start domain beaker-test-vm1.beaker error: operation forbidden: read only access prevents virDomainCreate
When I look at my laptop, there is no /var/run/libvirt/libvirt-sock. So.....I've been wondering whether RHEL7 virsh/libvirt is compatible with Fedora36. Is there a work-around? I can't change the distros on my laptop or VMs.
Hello, since Fedora 35 libvirt has used the modular daemons architecture. What this means for you (and appears very confusing) is that when you try starting a VM locally on your F36 laptop, your virsh client doesn't connect to libvirt-sock anymore, there's a dedicated connection socket for each of the daemons now and instead will connect to virtqemud-sock. Now, old virsh clients like the one you have on your RHEL7 don't know about this and expect to connect to libvirt-sock instead. In order to create that socket and restore functionality for old clients you need to start and enable the virtproxyd.socket systemd unit which proxies old client connections to the new sockets we have. Why the virtproxyd socket isn't running by default unless you disabled it beats me, since:
$ systemctl status virtproxyd Loaded: loaded (/usr/lib/systemd/system/virtproxyd.socket; enabled; vendor preset: enabled) ^^^here^^^
is set correctly after installation. Anyhow, just do: $ sudo systemctl enable --now virtproxyd.socket
on your laptop and you're good to go
Regards, Erik
So, if you don't have the libvirt-sock created that means the virtproxyd.socket systemd unit isn't active. Just enable the socket and try again.
Here, I simulated it for you with my VMs:
VM1: $ cat /etc/os-release NAME="Fedora Linux" VERSION="36 (Thirty Six)" ...
Uhm, sorry for the noise in ^this very last paragraph, it's a leftover draft of my response... Erik
On Wed, Jul 27, 2022 at 01:18:00PM -0400, Carol Bouchard wrote:
I have a Fedora36 laptop which hosts VMs with RHEL7 using libvirt. One of the RHEL7 VMs, runs remote commands (as root) to 'start' another VM by way of my laptop. In other words, the following command is run: virsh --connect 'qemu+ssh://192.168.120.1/system' start beaker-test-vm1.beaker If I run non-remote version of the command on the laptop, it is successful. For example, virsh --connect qemu:///system start beaker-test-vm1.beaker <-- Successful on laptop.
If I do a query like the following *(notice socket use)*, it is successful. virsh -d0 --connect 'qemu+ssh://192.168.120.1/system?*socket*=/var/run/libvirt/libvirt-sock-ro' domstate beaker-test-vm1.beaker
Without socket, I get the following error:
*error: failed to connect to the hypervisor*
*error: End of file while reading data: Ncat: No such file or directory.: Input/output error*
This is peculiar, it suggests that /var/run/libvirt/libvirt-sock does not exist, while /var/run/libvirt/libvirt-sock-ro does exist. This ought to be an impossible situation in general. As Erik says, In Fedora 36 we have the moduler daemons, so these two sockets are provided by 'virtproxyd.socket' and 'virtproxyd-ro.socket' unit files, so make sure both of those are running. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
TY VM!!! virtproxyd was disabled and I can assure you I didn't disable it. /run/libvirt/libvirt-sock now exists AND the remote virsh actions were successful. Background on my fedora36 install. I did not do an upgrade. This was a fresh install on a new laptop. I could see libvirt was running so I assumed it was intact. I had enabled/disabled libvirtd only because my remote virsh commands were not working. BTW I can't do the ssh approach as the scripts are used by other teams and I could break them. There had to be a better solution and use of proxy is an good one. Carol On Fri, Jul 29, 2022 at 4:43 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Wed, Jul 27, 2022 at 01:18:00PM -0400, Carol Bouchard wrote:
I have a Fedora36 laptop which hosts VMs with RHEL7 using libvirt. One of the RHEL7 VMs, runs remote commands (as root) to 'start' another VM by way of my laptop. In other words, the following command is run: virsh --connect 'qemu+ssh://192.168.120.1/system' start beaker-test-vm1.beaker If I run non-remote version of the command on the laptop, it is successful. For example, virsh --connect qemu:///system start beaker-test-vm1.beaker <-- Successful on laptop.
If I do a query like the following *(notice socket use)*, it is successful. virsh -d0 --connect 'qemu+ssh:// 192.168.120.1/system?*socket*=/var/run/libvirt/libvirt-sock-ro' domstate beaker-test-vm1.beaker
Without socket, I get the following error:
*error: failed to connect to the hypervisor*
*error: End of file while reading data: Ncat: No such file or directory.: Input/output error*
This is peculiar, it suggests that
/var/run/libvirt/libvirt-sock
does not exist, while /var/run/libvirt/libvirt-sock-ro does exist.
This ought to be an impossible situation in general. As Erik says, In Fedora 36 we have the moduler daemons, so these two sockets are provided by 'virtproxyd.socket' and 'virtproxyd-ro.socket' unit files, so make sure both of those are running.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Jul 29, 2022 at 07:49:16AM -0400, Carol Bouchard wrote:
TY VM!!! virtproxyd was disabled and I can assure you I didn't disable it.
/run/libvirt/libvirt-sock now exists AND the remote virsh actions were successful.
Background on my fedora36 install. I did not do an upgrade. This was a fresh install on a new laptop. I could see libvirt was running so I assumed it was intact. I had enabled/disabled libvirtd only because my remote virsh commands were not working.
Enabling/disabling libvirtd probably interfered with virtproxyd, as they both want the same sockets. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
The only reason I toggled libvirtd was because the remote virsh commands failed and I could see the socket didn't exist. That suggests to me that virtproxyd wasn't up AND it was configured at disabled. Your help was extremely appreciated! My test tool works now. So now I can test outside customer environment. This test tool is very important to me and my customers so I don't break them. Carol Carol On Fri, Jul 29, 2022 at 8:02 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Fri, Jul 29, 2022 at 07:49:16AM -0400, Carol Bouchard wrote:
TY VM!!! virtproxyd was disabled and I can assure you I didn't disable it.
/run/libvirt/libvirt-sock now exists AND the remote virsh actions were successful.
Background on my fedora36 install. I did not do an upgrade. This was a fresh install on a new laptop. I could see libvirt was running so I assumed it was intact. I had enabled/disabled libvirtd only because my remote virsh commands were not working.
Enabling/disabling libvirtd probably interfered with virtproxyd, as they both want the same sockets.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Fri, Jul 29, 2022 at 08:19:04AM -0400, Carol Bouchard wrote:
The only reason I toggled libvirtd was because the remote virsh commands failed and I could see the socket didn't exist. That suggests to me that virtproxyd wasn't up AND it was configured at disabled.
Note, that it is important to distinguish the service from the sockets. In Fedora 36, not only are we using modular daemons by default, but we also use socket activation. In a fresh install, you should only see the following services enabled virtqemud.service virtxend.service virtlxcd.service Those are needed becuase of requiremenmt todo VM auto-start on host bootup. For everything else we only expect sockets enabled: virtproxyd.socket virtinterfaced.socket virtnetworkd.socket virtnodedevd.socket virtnwfilterd.socket virtsecretd.socket virtstoraged.socket IOW, virtproxyd.service should not be enabled - it is autostarted when someone connects to virtproxyd.socket. And nothing related to libvirtd.service or libvirtd.socket should be enabled With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
These services do not exist [cbouchar@ibm-p8-rhevm-17-fsp cbouchar-devel]$ systemctl status virtxend.service Unit virtxend.service could not be found. [cbouchar@ibm-p8-rhevm-17-fsp cbouchar-devel]$ systemctl status virtxcd.service Unit virtxcd.service could not be found. On Mon, Aug 1, 2022 at 7:18 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Fri, Jul 29, 2022 at 08:19:04AM -0400, Carol Bouchard wrote:
The only reason I toggled libvirtd was because the remote virsh commands failed and I could see the socket didn't exist. That suggests to me that virtproxyd wasn't up AND it was configured at disabled.
Note, that it is important to distinguish the service from the sockets.
In Fedora 36, not only are we using modular daemons by default, but we also use socket activation. In a fresh install, you should only see the following services enabled
virtqemud.service virtxend.service virtlxcd.service
Those are needed becuase of requiremenmt todo VM auto-start on host bootup.
For everything else we only expect sockets enabled:
virtproxyd.socket virtinterfaced.socket virtnetworkd.socket virtnodedevd.socket virtnwfilterd.socket virtsecretd.socket virtstoraged.socket
IOW, virtproxyd.service should not be enabled - it is autostarted when someone connects to virtproxyd.socket.
And nothing related to libvirtd.service or libvirtd.socket should be enabled
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Aug 01, 2022 at 07:42:51AM -0400, Carol Bouchard wrote:
These services do not exist
[cbouchar@ibm-p8-rhevm-17-fsp cbouchar-devel]$ systemctl status virtxend.service Unit virtxend.service could not be found. [cbouchar@ibm-p8-rhevm-17-fsp cbouchar-devel]$ systemctl status virtxcd.service Unit virtxcd.service could not be found.
That's fine if you've not installed that piece of functionality. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
I believe I've found what has messed things up. I've run beaker-in-a-box ansible scripts which includes the following role: https://github.com/beaker-project/beaker-in-a-box/blob/master/roles/libvirt/... as well as a few other changes. So I found these instructions for installing libvirt but it seems obsolete for fedora36 https://computingforgeeks.com/how-to-install-kvm-on-fedora/ Do you have a list of install steps for the various fedoras? I do have things working with info you and Martin Kletzander provided me but would like to see a formal list in case there is something else that comes up. Carol On Mon, Aug 1, 2022 at 7:48 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Mon, Aug 01, 2022 at 07:42:51AM -0400, Carol Bouchard wrote:
These services do not exist
[cbouchar@ibm-p8-rhevm-17-fsp cbouchar-devel]$ systemctl status virtxend.service Unit virtxend.service could not be found. [cbouchar@ibm-p8-rhevm-17-fsp cbouchar-devel]$ systemctl status virtxcd.service Unit virtxcd.service could not be found.
That's fine if you've not installed that piece of functionality.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Mon, Aug 01, 2022 at 08:41:02AM -0400, Carol Bouchard wrote:
I believe I've found what has messed things up. I've run beaker-in-a-box ansible scripts which includes the following role: https://github.com/beaker-project/beaker-in-a-box/blob/master/roles/libvirt/... as well as a few other changes.
That needs to be changed to not touch the .services at all, and instead deal with .sockets. And also not libvirtd, for RHEL >= 9 or Fedora >= 35
So I found these instructions for installing libvirt but it seems obsolete for fedora36 https://computingforgeeks.com/how-to-install-kvm-on-fedora/ Do you have a list of install steps for the various fedoras?
The install steps haven't changed. All that's different is what systemd units are active by default. For a fresh install the right thing should "just work" out of the box, due to systemd presets. The problem is this ansible script is trying to manually start systemd services, which ignores the fact they shouldn't be running, as they are socket activated, and also doesnt' take into account libvirtd is no longer used on new Fedora/RHEL
I do have things working with info you and Martin Kletzander provided me but would like to see a formal list in case there is something else that comes up.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Daniel: Thank you very much for your feedback! Carol On Mon, Aug 1, 2022 at 9:05 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Mon, Aug 01, 2022 at 08:41:02AM -0400, Carol Bouchard wrote:
I believe I've found what has messed things up. I've run beaker-in-a-box ansible scripts which includes the following role:
https://github.com/beaker-project/beaker-in-a-box/blob/master/roles/libvirt/...
as well as a few other changes.
That needs to be changed to not touch the .services at all, and instead deal with .sockets. And also not libvirtd, for RHEL >= 9 or Fedora >= 35
So I found these instructions for installing libvirt but it seems
obsolete
for fedora36 https://computingforgeeks.com/how-to-install-kvm-on-fedora/ Do you have a list of install steps for the various fedoras?
The install steps haven't changed. All that's different is what systemd units are active by default. For a fresh install the right thing should "just work" out of the box, due to systemd presets.
The problem is this ansible script is trying to manually start systemd services, which ignores the fact they shouldn't be running, as they are socket activated, and also doesnt' take into account libvirtd is no longer used on new Fedora/RHEL
I do have things working with info you and Martin Kletzander provided me but would like to see a formal list in case there is something else that comes up.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (4)
-
Benjamin Smith -
Carol Bouchard -
Daniel P. Berrangé -
Erik Skultety