per user vm isolation with shared network
Greetings, I have two vm which I want to isolate per user, if I'm not mistaken, I can to that with per session uri. but I want to setup a virtual bridge so they will get connected with each other. looks like that if I define the network as system, it isn't visible in the session. is there a way to do that? if I define the same network in both sessions, will it work? Thanks, Dagg
On 6/30/24 01:01, daggs via Users wrote:
Greetings,
I have two vm which I want to isolate per user, if I'm not mistaken, I can to that with per session uri. but I want to setup a virtual bridge so they will get connected with each other. looks like that if I define the network as system, it isn't visible in the session. is there a way to do that? if I define the same network in both sessions, will it work?
Thanks,
Dagg
Yeah, this is known issue: https://gitlab.com/libvirt/libvirt/-/issues/438 what you can do is create a bridge and then use qemu-bridge-helper to plug TAPs from qemu:///session VMs into the bridge. Theoretically, you could even misuse virbr0. Michal
Greetings,
Sent: Monday, July 01, 2024 at 10:35 AM From: "Michal Prívozník" <mprivozn@redhat.com> To: "daggs" <daggs@gmx.com>, users@lists.libvirt.org Subject: Re: per user vm isolation with shared network
On 6/30/24 01:01, daggs via Users wrote:
Greetings,
I have two vm which I want to isolate per user, if I'm not mistaken, I can to that with per session uri. but I want to setup a virtual bridge so they will get connected with each other. looks like that if I define the network as system, it isn't visible in the session. is there a way to do that? if I define the same network in both sessions, will it work?
Thanks,
Dagg
Yeah, this is known issue:
https://gitlab.com/libvirt/libvirt/-/issues/438
what you can do is create a bridge and then use qemu-bridge-helper to plug TAPs from qemu:///session VMs into the bridge. Theoretically, you could even misuse virbr0.
Michal
thanks, I've started looking into it however I'm unable to see the network from the dedicated user, see: $ id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) Name State Autostart Persistent -------------------------------------------- default active yes yes Name State Autostart Persistent -------------------------------------------- default active yes yes $ su -c "id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all" foo uid=1002(foo) gid=1002(foo) groups=1002(foo),34(kvm),36(qemu),102(libvirt) Name State Autostart Persistent ---------------------------------------- error: failed to connect to the hypervisor error: internal error: Unable to get system bus connection: Could not connect: No such file or directory $ cat /etc/qemu/bridge.conf # This should have the following permissions: root:qemu 0640 # Allow users in the "qemu" group to add devices to "br0". allow br0 # Uncomment the following line to allow users in the "bob" # group to have permissions defined in it, iff it has the # following permissions: root:bob 0640 #include /etc/qemu/bob.conf $ brctl show virbr0 bridge name bridge id STP enabled interfaces virbr0 8000.525400892227 yes utils-server:/home/igor$ ls -l /usr/lib/qemu/qemu-bridge-helper -rws--x--- 1 root qemu 837704 Jun 18 14:07 /usr/lib/qemu/qemu-bridge-helper utils-server:/home/igor$ virsh -c qemu:///session net-dumpxml default <network> <name>default</name> <uuid>9ec5f56a-1c3d-4650-aac2-45d1352e08fc</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:89:22:27'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network> what am I missing?
On 7/1/24 12:28, daggs wrote:
Greetings,
Sent: Monday, July 01, 2024 at 10:35 AM From: "Michal Prívozník" <mprivozn@redhat.com> To: "daggs" <daggs@gmx.com>, users@lists.libvirt.org Subject: Re: per user vm isolation with shared network
On 6/30/24 01:01, daggs via Users wrote:
Greetings,
I have two vm which I want to isolate per user, if I'm not mistaken, I can to that with per session uri. but I want to setup a virtual bridge so they will get connected with each other. looks like that if I define the network as system, it isn't visible in the session. is there a way to do that? if I define the same network in both sessions, will it work?
Thanks,
Dagg
Yeah, this is known issue:
https://gitlab.com/libvirt/libvirt/-/issues/438
what you can do is create a bridge and then use qemu-bridge-helper to plug TAPs from qemu:///session VMs into the bridge. Theoretically, you could even misuse virbr0.
Michal
thanks, I've started looking into it however I'm unable to see the network from the dedicated user, see: $ id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) Name State Autostart Persistent -------------------------------------------- default active yes yes
Name State Autostart Persistent -------------------------------------------- default active yes yes
$ su -c "id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all" foo uid=1002(foo) gid=1002(foo) groups=1002(foo),34(kvm),36(qemu),102(libvirt) Name State Autostart Persistent ----------------------------------------
error: failed to connect to the hypervisor error: internal error: Unable to get system bus connection: Could not connect: No such file or directory
This is expected and in fact it's what the issue I've linked earlier is all about.
$ cat /etc/qemu/bridge.conf # This should have the following permissions: root:qemu 0640
# Allow users in the "qemu" group to add devices to "br0". allow br0
put "allow virbr0" here. And then have your qemu:///session domain use: <interface type='bridge'> <source bridge='virbr0'/> </interface> OR, if you don't want to use virbr0 from qemu:///system you can s/virbr0/br0/ in the XML snippet and create br0 yourself. virbr0 has NAT which is something you may not want. Michal
Sent: Monday, July 01, 2024 at 2:32 PM From: "Michal Prívozník" <mprivozn@redhat.com> To: "daggs" <daggs@gmx.com> Cc: users@lists.libvirt.org Subject: Re: per user vm isolation with shared network
On 7/1/24 12:28, daggs wrote:
Greetings,
Sent: Monday, July 01, 2024 at 10:35 AM From: "Michal Prívozník" <mprivozn@redhat.com> To: "daggs" <daggs@gmx.com>, users@lists.libvirt.org Subject: Re: per user vm isolation with shared network
On 6/30/24 01:01, daggs via Users wrote:
Greetings,
I have two vm which I want to isolate per user, if I'm not mistaken, I can to that with per session uri. but I want to setup a virtual bridge so they will get connected with each other. looks like that if I define the network as system, it isn't visible in the session. is there a way to do that? if I define the same network in both sessions, will it work?
Thanks,
Dagg
Yeah, this is known issue:
https://gitlab.com/libvirt/libvirt/-/issues/438
what you can do is create a bridge and then use qemu-bridge-helper to plug TAPs from qemu:///session VMs into the bridge. Theoretically, you could even misuse virbr0.
Michal
thanks, I've started looking into it however I'm unable to see the network from the dedicated user, see: $ id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) Name State Autostart Persistent -------------------------------------------- default active yes yes
Name State Autostart Persistent -------------------------------------------- default active yes yes
$ su -c "id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all" foo uid=1002(foo) gid=1002(foo) groups=1002(foo),34(kvm),36(qemu),102(libvirt) Name State Autostart Persistent ----------------------------------------
error: failed to connect to the hypervisor error: internal error: Unable to get system bus connection: Could not connect: No such file or directory
This is expected and in fact it's what the issue I've linked earlier is all about.
$ cat /etc/qemu/bridge.conf # This should have the following permissions: root:qemu 0640
# Allow users in the "qemu" group to add devices to "br0". allow br0
put "allow virbr0" here.
And then have your qemu:///session domain use:
<interface type='bridge'> <source bridge='virbr0'/> </interface>
OR, if you don't want to use virbr0 from qemu:///system you can s/virbr0/br0/ in the XML snippet and create br0 yourself. virbr0 has NAT which is something you may not want.
Michal
thanks, seems like I'm past this part, the vm start fails because of insufficient permissions to detach/reattach the pci nodes, I assumed that there is no fast solution so I fixed it with a script that uses doas to preform the detach/reattach. now I get this error: error: Unable to create tap device vnet0: Operation not permitted seems like I need special permissions to create the tap device. to I need to create the tap device and attach it to the bridge as part of the prepare stage?
On 7/1/24 21:55, daggs via Users wrote:
Sent: Monday, July 01, 2024 at 2:32 PM From: "Michal Prívozník" <mprivozn@redhat.com> To: "daggs" <daggs@gmx.com> Cc: users@lists.libvirt.org Subject: Re: per user vm isolation with shared network
On 7/1/24 12:28, daggs wrote:
Greetings,
Sent: Monday, July 01, 2024 at 10:35 AM From: "Michal Prívozník" <mprivozn@redhat.com> To: "daggs" <daggs@gmx.com>, users@lists.libvirt.org Subject: Re: per user vm isolation with shared network
On 6/30/24 01:01, daggs via Users wrote:
Greetings,
I have two vm which I want to isolate per user, if I'm not mistaken, I can to that with per session uri. but I want to setup a virtual bridge so they will get connected with each other. looks like that if I define the network as system, it isn't visible in the session. is there a way to do that? if I define the same network in both sessions, will it work?
Thanks,
Dagg
Yeah, this is known issue:
https://gitlab.com/libvirt/libvirt/-/issues/438
what you can do is create a bridge and then use qemu-bridge-helper to plug TAPs from qemu:///session VMs into the bridge. Theoretically, you could even misuse virbr0.
Michal
thanks, I've started looking into it however I'm unable to see the network from the dedicated user, see: $ id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) Name State Autostart Persistent -------------------------------------------- default active yes yes
Name State Autostart Persistent -------------------------------------------- default active yes yes
$ su -c "id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all" foo uid=1002(foo) gid=1002(foo) groups=1002(foo),34(kvm),36(qemu),102(libvirt) Name State Autostart Persistent ----------------------------------------
error: failed to connect to the hypervisor error: internal error: Unable to get system bus connection: Could not connect: No such file or directory
This is expected and in fact it's what the issue I've linked earlier is all about.
$ cat /etc/qemu/bridge.conf # This should have the following permissions: root:qemu 0640
# Allow users in the "qemu" group to add devices to "br0". allow br0
put "allow virbr0" here.
And then have your qemu:///session domain use:
<interface type='bridge'> <source bridge='virbr0'/> </interface>
OR, if you don't want to use virbr0 from qemu:///system you can s/virbr0/br0/ in the XML snippet and create br0 yourself. virbr0 has NAT which is something you may not want.
Michal
thanks, seems like I'm past this part, the vm start fails because of insufficient permissions to detach/reattach the pci nodes, I assumed that there is no fast solution so I fixed it with a script that uses doas to preform the detach/reattach.
No sysadmin wants to allow regular users to bind PCI devices to "random" drivers, surely. PCI devices must be "detached" (i.e. bound to vfio driver) by sysadmin (e.g. virsh -c qemu:///system nodedev-detach ...) BEFORE qemu:///session domain wants to use the device.
now I get this error: error: Unable to create tap device vnet0: Operation not permitted
seems like I need special permissions to create the tap device. to I need to create the tap device and attach it to the bridge as part of the prepare stage?
I'm not sure why your system refuses that. Works on my machine just fine. You should talk to your distro vendor about that decission. Michal
Sent: Tuesday, July 02, 2024 at 3:59 PM From: "Michal Prívozník" <mprivozn@redhat.com> To: "daggs" <daggs@gmx.com> Cc: users@lists.libvirt.org Subject: Re: per user vm isolation with shared network
On 7/1/24 21:55, daggs via Users wrote:
Sent: Monday, July 01, 2024 at 2:32 PM From: "Michal Prívozník" <mprivozn@redhat.com> To: "daggs" <daggs@gmx.com> Cc: users@lists.libvirt.org Subject: Re: per user vm isolation with shared network
On 7/1/24 12:28, daggs wrote:
Greetings,
Sent: Monday, July 01, 2024 at 10:35 AM From: "Michal Prívozník" <mprivozn@redhat.com> To: "daggs" <daggs@gmx.com>, users@lists.libvirt.org Subject: Re: per user vm isolation with shared network
On 6/30/24 01:01, daggs via Users wrote:
Greetings,
I have two vm which I want to isolate per user, if I'm not mistaken, I can to that with per session uri. but I want to setup a virtual bridge so they will get connected with each other. looks like that if I define the network as system, it isn't visible in the session. is there a way to do that? if I define the same network in both sessions, will it work?
Thanks,
Dagg
Yeah, this is known issue:
https://gitlab.com/libvirt/libvirt/-/issues/438
what you can do is create a bridge and then use qemu-bridge-helper to plug TAPs from qemu:///session VMs into the bridge. Theoretically, you could even misuse virbr0.
Michal
thanks, I've started looking into it however I'm unable to see the network from the dedicated user, see: $ id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) Name State Autostart Persistent -------------------------------------------- default active yes yes
Name State Autostart Persistent -------------------------------------------- default active yes yes
$ su -c "id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all" foo uid=1002(foo) gid=1002(foo) groups=1002(foo),34(kvm),36(qemu),102(libvirt) Name State Autostart Persistent ----------------------------------------
error: failed to connect to the hypervisor error: internal error: Unable to get system bus connection: Could not connect: No such file or directory
This is expected and in fact it's what the issue I've linked earlier is all about.
$ cat /etc/qemu/bridge.conf # This should have the following permissions: root:qemu 0640
# Allow users in the "qemu" group to add devices to "br0". allow br0
put "allow virbr0" here.
And then have your qemu:///session domain use:
<interface type='bridge'> <source bridge='virbr0'/> </interface>
OR, if you don't want to use virbr0 from qemu:///system you can s/virbr0/br0/ in the XML snippet and create br0 yourself. virbr0 has NAT which is something you may not want.
Michal
thanks, seems like I'm past this part, the vm start fails because of insufficient permissions to detach/reattach the pci nodes, I assumed that there is no fast solution so I fixed it with a script that uses doas to preform the detach/reattach.
No sysadmin wants to allow regular users to bind PCI devices to "random" drivers, surely. PCI devices must be "detached" (i.e. bound to vfio driver) by sysadmin (e.g. virsh -c qemu:///system nodedev-detach ...) BEFORE qemu:///session domain wants to use the device. so I should call virsh -c qemu:///system nodedev-detach from within the libvirt hook? wont that might cause a hangup? I did got that in some scenarios.
now I get this error: error: Unable to create tap device vnet0: Operation not permitted
seems like I need special permissions to create the tap device. to I need to create the tap device and attach it to the bridge as part of the prepare stage?
I'm not sure why your system refuses that. Works on my machine just fine. You should talk to your distro vendor about that decission.
Michal
will do, thanks
On 7/2/24 16:19, daggs wrote:
thanks, seems like I'm past this part, the vm start fails because of insufficient permissions to detach/reattach the pci nodes, I assumed that there is no fast solution so I fixed it with a script that uses doas to preform the detach/reattach.
No sysadmin wants to allow regular users to bind PCI devices to "random" drivers, surely. PCI devices must be "detached" (i.e. bound to vfio driver) by sysadmin (e.g. virsh -c qemu:///system nodedev-detach ...) BEFORE qemu:///session domain wants to use the device. so I should call virsh -c qemu:///system nodedev-detach from within the libvirt hook? wont that might cause a hangup? I did got that in some scenarios.
No, calling libvirt from hooks is strongly discouraged as deadlocks are likely to occur. Just detach PCI devices before starting any qemu:///session domain. Either right at startup (write an init service), do that manually, doesn't matter really. Michal
Sent: Tuesday, July 02, 2024 at 5:25 PM From: "Michal Prívozník" <mprivozn@redhat.com> To: "daggs" <daggs@gmx.com> Cc: users@lists.libvirt.org Subject: Re: per user vm isolation with shared network
On 7/2/24 16:19, daggs wrote:
thanks, seems like I'm past this part, the vm start fails because of insufficient permissions to detach/reattach the pci nodes, I assumed that there is no fast solution so I fixed it with a script that uses doas to preform the detach/reattach.
No sysadmin wants to allow regular users to bind PCI devices to "random" drivers, surely. PCI devices must be "detached" (i.e. bound to vfio driver) by sysadmin (e.g. virsh -c qemu:///system nodedev-detach ...) BEFORE qemu:///session domain wants to use the device. so I should call virsh -c qemu:///system nodedev-detach from within the libvirt hook? wont that might cause a hangup? I did got that in some scenarios.
No, calling libvirt from hooks is strongly discouraged as deadlocks are likely to occur. Just detach PCI devices before starting any qemu:///session domain. Either right at startup (write an init service), do that manually, doesn't matter really.
Michal
that seems more complicated as I want to be sure that all devs are reattached on termination, so I cannot use autostart as I need to detach the devs prior to starting it from the init script. I can add a test to see if the vm was started via the init script, if not error. I can use atd daemon to call run the reattachment from the hook but depend on the condition that virsh was terminated. regarding the error I got, can you point me to the location in the code that preforms it? waiting for the distro's devs might take too long so I want to try and solve it by myself
participants (2)
-
daggs -
Michal Prívozník