Sent: Monday, July 01, 2024 at 10:35 AM From: "Michal Prívozník" <mprivozn(a)redhat.com&gt; To: "daggs" <daggs(a)gmx.com&gt;, users(a)lists.libvirt.org Subject: Re: per user vm isolation with shared network On 6/30/24 01:01, daggs via Users wrote: > Greetings, > > I have two vm which I want to isolate per user, if I'm not mistaken, I can to that with per session uri. > but I want to setup a virtual bridge so they will get connected with each other. > looks like that if I define the network as system, it isn't visible in the session. > is there a way to do that? if I define the same network in both sessions, will it work? > > Thanks, > > Dagg > Yeah, this is known issue: https://gitlab.com/libvirt/libvirt/-/issues/438 what you can do is create a bridge and then use qemu-bridge-helper to plug TAPs from qemu:///session VMs into the bridge. Theoretically, you could even misuse virbr0. Michal

Sent: Monday, July 01, 2024 at 2:32 PM From: "Michal Prívozník" <mprivozn(a)redhat.com&gt; To: "daggs" <daggs(a)gmx.com&gt; Cc: users(a)lists.libvirt.org Subject: Re: per user vm isolation with shared network On 7/1/24 12:28, daggs wrote: > Greetings, > >> Sent: Monday, July 01, 2024 at 10:35 AM >> From: "Michal Prívozník" <mprivozn(a)redhat.com&gt; >> To: "daggs" <daggs(a)gmx.com&gt;, users(a)lists.libvirt.org >> Subject: Re: per user vm isolation with shared network >> >> On 6/30/24 01:01, daggs via Users wrote: >>> Greetings, >>> >>> I have two vm which I want to isolate per user, if I'm not mistaken, I can to that with per session uri. >>> but I want to setup a virtual bridge so they will get connected with each other. >>> looks like that if I define the network as system, it isn't visible in the session. >>> is there a way to do that? if I define the same network in both sessions, will it work? >>> >>> Thanks, >>> >>> Dagg >>> >> >> Yeah, this is known issue: >> >> https://gitlab.com/libvirt/libvirt/-/issues/438 >> >> what you can do is create a bridge and then use qemu-bridge-helper to >> plug TAPs from qemu:///session VMs into the bridge. Theoretically, you >> could even misuse virbr0. >> >> Michal > > thanks, I've started looking into it however I'm unable to see the network from the dedicated user, see: > $ id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all > uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) > Name State Autostart Persistent > -------------------------------------------- > default active yes yes > > Name State Autostart Persistent > -------------------------------------------- > default active yes yes > > $ su -c "id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all" foo > uid=1002(foo) gid=1002(foo) groups=1002(foo),34(kvm),36(qemu),102(libvirt) > Name State Autostart Persistent > ---------------------------------------- > > error: failed to connect to the hypervisor > error: internal error: Unable to get system bus connection: Could not connect: No such file or directory This is expected and in fact it's what the issue I've linked earlier is all about. > $ cat /etc/qemu/bridge.conf > # This should have the following permissions: root:qemu 0640 > > # Allow users in the "qemu" group to add devices to "br0". > allow br0 put "allow virbr0" here. And then have your qemu:///session domain use: <interface type='bridge'> <source bridge='virbr0'/> </interface> OR, if you don't want to use virbr0 from qemu:///system you can s/virbr0/br0/ in the XML snippet and create br0 yourself. virbr0 has NAT which is something you may not want. Michal

Sent: Tuesday, July 02, 2024 at 3:59 PM From: "Michal Prívozník" <mprivozn(a)redhat.com&gt; To: "daggs" <daggs(a)gmx.com&gt; Cc: users(a)lists.libvirt.org Subject: Re: per user vm isolation with shared network On 7/1/24 21:55, daggs via Users wrote: >> Sent: Monday, July 01, 2024 at 2:32 PM >> From: "Michal Prívozník" <mprivozn(a)redhat.com&gt; >> To: "daggs" <daggs(a)gmx.com&gt; >> Cc: users(a)lists.libvirt.org >> Subject: Re: per user vm isolation with shared network >> >> On 7/1/24 12:28, daggs wrote: >>> Greetings, >>> >>>> Sent: Monday, July 01, 2024 at 10:35 AM >>>> From: "Michal Prívozník" <mprivozn(a)redhat.com&gt; >>>> To: "daggs" <daggs(a)gmx.com&gt;, users(a)lists.libvirt.org >>>> Subject: Re: per user vm isolation with shared network >>>> >>>> On 6/30/24 01:01, daggs via Users wrote: >>>>> Greetings, >>>>> >>>>> I have two vm which I want to isolate per user, if I'm not mistaken, I can to that with per session uri. >>>>> but I want to setup a virtual bridge so they will get connected with each other. >>>>> looks like that if I define the network as system, it isn't visible in the session. >>>>> is there a way to do that? if I define the same network in both sessions, will it work? >>>>> >>>>> Thanks, >>>>> >>>>> Dagg >>>>> >>>> >>>> Yeah, this is known issue: >>>> >>>> https://gitlab.com/libvirt/libvirt/-/issues/438 >>>> >>>> what you can do is create a bridge and then use qemu-bridge-helper to >>>> plug TAPs from qemu:///session VMs into the bridge. Theoretically, you >>>> could even misuse virbr0. >>>> >>>> Michal >>> >>> thanks, I've started looking into it however I'm unable to see the network from the dedicated user, see: >>> $ id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all >>> uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) >>> Name State Autostart Persistent >>> -------------------------------------------- >>> default active yes yes >>> >>> Name State Autostart Persistent >>> -------------------------------------------- >>> default active yes yes >>> >>> $ su -c "id; virsh -c qemu:///session net-list --all; virsh -c qemu:///system net-list --all" foo >>> uid=1002(foo) gid=1002(foo) groups=1002(foo),34(kvm),36(qemu),102(libvirt) >>> Name State Autostart Persistent >>> ---------------------------------------- >>> >>> error: failed to connect to the hypervisor >>> error: internal error: Unable to get system bus connection: Could not connect: No such file or directory >> >> This is expected and in fact it's what the issue I've linked earlier is >> all about. >> >>> $ cat /etc/qemu/bridge.conf >>> # This should have the following permissions: root:qemu 0640 >>> >>> # Allow users in the "qemu" group to add devices to "br0". >>> allow br0 >> >> put "allow virbr0" here. >> >> And then have your qemu:///session domain use: >> >> <interface type='bridge'> >> <source bridge='virbr0'/> >> </interface> >> >> OR, if you don't want to use virbr0 from qemu:///system you can >> s/virbr0/br0/ in the XML snippet and create br0 yourself. virbr0 has NAT >> which is something you may not want. >> >> Michal >> >> > > thanks, seems like I'm past this part, the vm start fails because of insufficient permissions to detach/reattach the pci nodes, I assumed that there is no fast solution so > I fixed it with a script that uses doas to preform the detach/reattach. No sysadmin wants to allow regular users to bind PCI devices to "random" drivers, surely. PCI devices must be "detached" (i.e. bound to vfio driver) by sysadmin (e.g. virsh -c qemu:///system nodedev-detach ...) BEFORE qemu:///session domain wants to use the device.

> now I get this error: error: Unable to create tap device vnet0: Operation not permitted > > seems like I need special permissions to create the tap device. > to I need to create the tap device and attach it to the bridge as part of the prepare stage? > I'm not sure why your system refuses that. Works on my machine just fine. You should talk to your distro vendor about that decission. Michal

Sent: Tuesday, July 02, 2024 at 5:25 PM From: "Michal Prívozník" <mprivozn(a)redhat.com&gt; To: "daggs" <daggs(a)gmx.com&gt; Cc: users(a)lists.libvirt.org Subject: Re: per user vm isolation with shared network On 7/2/24 16:19, daggs wrote: >>> thanks, seems like I'm past this part, the vm start fails because of insufficient permissions to detach/reattach the pci nodes, I assumed that there is no fast solution so >>> I fixed it with a script that uses doas to preform the detach/reattach. >> >> No sysadmin wants to allow regular users to bind PCI devices to "random" >> drivers, surely. PCI devices must be "detached" (i.e. bound to vfio >> driver) by sysadmin (e.g. virsh -c qemu:///system nodedev-detach ...) >> BEFORE qemu:///session domain wants to use the device. > so I should call virsh -c qemu:///system nodedev-detach from within the libvirt hook? wont that might cause a hangup? > I did got that in some scenarios. No, calling libvirt from hooks is strongly discouraged as deadlocks are likely to occur. Just detach PCI devices before starting any qemu:///session domain. Either right at startup (write an init service), do that manually, doesn't matter really. Michal