Gao feng <gaofeng(a)cn.fujitsu.com&gt; writes: > cc libvirt-list > > On 08/21/2013 01:30 PM, Eric W. Biederman wrote: >> Gao feng <gaofeng(a)cn.fujitsu.com&gt; writes: >> >>> Unix sockets are private resources of net namespace, >>> allowing one net namespace to access to other netns's unix >>> sockets is meaningless. >> >> Allowing one net namespace to access another netns's unix socket is >> deliberate behavior. This is a desired and useful feature, and >> only a misconfiguration of visible files would allow this to be a >> problem. >> >>> I'm researching a problem about shutdown from container, >>> if the cotainer shares the same file /run/systemd/private >>> with host, when we run shutdown -h xxx in container, the >>> shutdown message will be send to the systemd-shutdownd >>> through unix socket /run/systemd/private, and because >>> systemd-shutdownd is running in host, so finally, the host >>> will become shutdown. >> >> The simple answer is don't do that then. I can see no reason >> to share /run outside of the container unless you want this kind of >> behavior. >> >> Quite frankly I want this behavior if I am using network namespaces >> to support multiple routing contexts. That is if I am using scripts >> like: >> >> ip netns add other >> ip netns exec other script >> >> I don't want to have to remember to say >> ip netns orig exec shutdown -h now >> >> There are more compelling uses and there is no cost in supporting this >> in the kernel. >> >> What kind of misconfiguration caused someone to complain about this? >> > > libvirt lxc allows user to set up a container which shares the same root > directory with host. > > seems like the unix sockets whose sun_path is an abstract socket address > are net namespace aware. > > Should we use "abstract" type of address instead of a file system pathname > for systemd in this case? I suspect libvirt should simply not share /run or any other normally writable directory with the host. Sharing /run /var/run or even /tmp seems extremely dubious if you want some kind of containment, and without strange things spilling through.

On Wed, Aug 21, 2013 at 9:22 AM, Gao feng <gaofeng(a)cn.fujitsu.com&gt; wrote: > On 08/21/2013 03:06 PM, Eric W. Biederman wrote: >> I suspect libvirt should simply not share /run or any other normally >> writable directory with the host. Sharing /run /var/run or even /tmp >> seems extremely dubious if you want some kind of containment, and >> without strange things spilling through. Right, /run or /var cannot be shared. It's not only about sockets, many other things will also go really wrong that way.

On Wed, 2013-08-21 at 11:51 +0200, Kay Sievers wrote: > On Wed, Aug 21, 2013 at 9:22 AM, Gao feng <gaofeng(a)cn.fujitsu.com&gt; wrote: > > On 08/21/2013 03:06 PM, Eric W. Biederman wrote: > > >> I suspect libvirt should simply not share /run or any other normally > >> writable directory with the host. Sharing /run /var/run or even /tmp > >> seems extremely dubious if you want some kind of containment, and > >> without strange things spilling through. > > Right, /run or /var cannot be shared. It's not only about sockets, > many other things will also go really wrong that way. This is very narrow thinking about what a container might be and will cause trouble as people start to create novel uses for containers in the cloud if you try to impose this on our current infrastructure. One of the cgroup only container uses we see at Parallels (so no separate filesystem and no net namespaces) is pure apache load balancer type shared hosting. In this scenario, base apache is effectively brought up in the host environment, but then spawned instances are resource limited using cgroups according to what the customer has paid. Obviously all apache instances are sharing /var and /run from the host (mostly for logging and pid storage and static pages). The reason some hosters do this is that it allows much higher density simple web serving (either static pages from quota limited chroots or dynamic pages limited by database space constraints) because each "instance" shares so much from the host. The service is obviously much more basic than giving each customer a container running apache, but it's much easier for the hoster to administer and it serves the customer just as well for a large cross section of use cases and for those it doesn't serve, the hoster usually has separate container hosting (for a higher price, of course).

On Sun, 2013-08-25 at 19:37 +0200, Kay Sievers wrote: > On Sun, Aug 25, 2013 at 7:16 PM, James Bottomley > <jbottomley(a)parallels.com&gt; wrote: >> On Wed, 2013-08-21 at 11:51 +0200, Kay Sievers wrote: >>> On Wed, Aug 21, 2013 at 9:22 AM, Gao feng <gaofeng(a)cn.fujitsu.com&gt; wrote: >>>> On 08/21/2013 03:06 PM, Eric W. Biederman wrote: >>> >>>>> I suspect libvirt should simply not share /run or any other normally >>>>> writable directory with the host. Sharing /run /var/run or even /tmp >>>>> seems extremely dubious if you want some kind of containment, and >>>>> without strange things spilling through. >>> >>> Right, /run or /var cannot be shared. It's not only about sockets, >>> many other things will also go really wrong that way. >> >> This is very narrow thinking about what a container might be and will >> cause trouble as people start to create novel uses for containers in the >> cloud if you try to impose this on our current infrastructure. >> >> One of the cgroup only container uses we see at Parallels (so no >> separate filesystem and no net namespaces) is pure apache load balancer >> type shared hosting. In this scenario, base apache is effectively >> brought up in the host environment, but then spawned instances are >> resource limited using cgroups according to what the customer has paid. >> Obviously all apache instances are sharing /var and /run from the host >> (mostly for logging and pid storage and static pages). The reason some >> hosters do this is that it allows much higher density simple web serving >> (either static pages from quota limited chroots or dynamic pages limited >> by database space constraints) because each "instance" shares so much >> from the host. The service is obviously much more basic than giving >> each customer a container running apache, but it's much easier for the >> hoster to administer and it serves the customer just as well for a large >> cross section of use cases and for those it doesn't serve, the hoster >> usually has separate container hosting (for a higher price, of course). > > The "container" as we talk about has it's own init, and no, it cannot > share /var or /run. This is what we would call an IaaS container: bringing up init and effectively a new OS inside a container is the closest containers come to being like hypervisors. It's the most common use case of Parallels containers in the field, so I'm certainly not telling you it's a bad idea. > The stuff you talk about has nothing to do with that, it's not > different from all services or a multi-instantiated service on the > host sharing the same /run and /var. I gave you one example: a really simplistic one. A more sophisticated example is a PaaS or SaaS container where you bring the OS up in the host but spawn a particular application into its own container (this is essentially similar to what Docker does). Often in this case, you do add separate mount and network namespaces to make the application isolated and migrateable with its own IP address. The reason you share init and most of the OS from the host is for elasticity and density, which are fast becoming a holy grail type quest of cloud orchestration systems: if you don't have to bring up the OS from init and you can just start the application from a C/R image (orders of magnitude smaller than a full system image) and slap on the necessary namespaces as you clone it, you have something that comes online in miliseconds which is a feat no hypervisor based virtualisation can match. I'm not saying don't pursue the IaaS case, it's definitely useful ... I'm just saying it would be a serious mistake to think that's the only use case for containers and we certainly shouldn't adjust Linux to serve only that use case.

On Mon, 2013-08-26 at 09:06 +0800, Gao feng wrote: > On 08/26/2013 02:16 AM, James Bottomley wrote: >> On Sun, 2013-08-25 at 19:37 +0200, Kay Sievers wrote: >>> On Sun, Aug 25, 2013 at 7:16 PM, James Bottomley >>> <jbottomley(a)parallels.com&gt; wrote: >>>> On Wed, 2013-08-21 at 11:51 +0200, Kay Sievers wrote: >>>>> On Wed, Aug 21, 2013 at 9:22 AM, Gao feng <gaofeng(a)cn.fujitsu.com&gt; wrote: >>>>>> On 08/21/2013 03:06 PM, Eric W. Biederman wrote: >>>>> >>>>>>> I suspect libvirt should simply not share /run or any other normally >>>>>>> writable directory with the host. Sharing /run /var/run or even /tmp >>>>>>> seems extremely dubious if you want some kind of containment, and >>>>>>> without strange things spilling through. >>>>> >>>>> Right, /run or /var cannot be shared. It's not only about sockets, >>>>> many other things will also go really wrong that way. >>>> >>>> This is very narrow thinking about what a container might be and will >>>> cause trouble as people start to create novel uses for containers in the >>>> cloud if you try to impose this on our current infrastructure. >>>> >>>> One of the cgroup only container uses we see at Parallels (so no >>>> separate filesystem and no net namespaces) is pure apache load balancer >>>> type shared hosting. In this scenario, base apache is effectively >>>> brought up in the host environment, but then spawned instances are >>>> resource limited using cgroups according to what the customer has paid. >>>> Obviously all apache instances are sharing /var and /run from the host >>>> (mostly for logging and pid storage and static pages). The reason some >>>> hosters do this is that it allows much higher density simple web serving >>>> (either static pages from quota limited chroots or dynamic pages limited >>>> by database space constraints) because each "instance" shares so much >>>> from the host. The service is obviously much more basic than giving >>>> each customer a container running apache, but it's much easier for the >>>> hoster to administer and it serves the customer just as well for a large >>>> cross section of use cases and for those it doesn't serve, the hoster >>>> usually has separate container hosting (for a higher price, of course). >>> >>> The "container" as we talk about has it's own init, and no, it cannot >>> share /var or /run. >> >> This is what we would call an IaaS container: bringing up init and >> effectively a new OS inside a container is the closest containers come >> to being like hypervisors. It's the most common use case of Parallels >> containers in the field, so I'm certainly not telling you it's a bad >> idea. >> >>> The stuff you talk about has nothing to do with that, it's not >>> different from all services or a multi-instantiated service on the >>> host sharing the same /run and /var. >> >> I gave you one example: a really simplistic one. A more sophisticated >> example is a PaaS or SaaS container where you bring the OS up in the >> host but spawn a particular application into its own container (this is >> essentially similar to what Docker does). Often in this case, you do >> add separate mount and network namespaces to make the application >> isolated and migrateable with its own IP address. The reason you share >> init and most of the OS from the host is for elasticity and density, >> which are fast becoming a holy grail type quest of cloud orchestration >> systems: if you don't have to bring up the OS from init and you can just >> start the application from a C/R image (orders of magnitude smaller than >> a full system image) and slap on the necessary namespaces as you clone >> it, you have something that comes online in miliseconds which is a feat >> no hypervisor based virtualisation can match. >> >> I'm not saying don't pursue the IaaS case, it's definitely useful ... >> I'm just saying it would be a serious mistake to think that's the only >> use case for containers and we certainly shouldn't adjust Linux to serve >> only that use case. >> > > The feature you said above VS contianer-reboot-host bug, I prefer to > fix > the bug. What bug? > and this feature can be achieved even container unshares /run > directory > with host by default, for libvirt, user can set the container > configuration to > make the container shares the /run directory with host. > > I would like to say, the reboot from container bug is more urgent and > need > to be fixed. Are you talking about the old bug where trying to reboot an lxc container from within it would reboot the entire system?

has never suffered from that problem and I thought it was fixed upstream. I've not tested lxc tools, but the latest vzctl from the openvz website will bring up a container on the vanilla 3.9 kernel (provided you have USER_NS compiled in) can also be used to reboot the container, so I see no reason it wouldn't work for lxc as well.

On 08/26/2013 11:19 AM, James Bottomley wrote: > On Mon, 2013-08-26 at 09:06 +0800, Gao feng wrote: >> On 08/26/2013 02:16 AM, James Bottomley wrote: >>> On Sun, 2013-08-25 at 19:37 +0200, Kay Sievers wrote: >>>> On Sun, Aug 25, 2013 at 7:16 PM, James Bottomley >>>> <jbottomley(a)parallels.com&gt; wrote: >>>>> On Wed, 2013-08-21 at 11:51 +0200, Kay Sievers wrote: >>>>>> On Wed, Aug 21, 2013 at 9:22 AM, Gao feng <gaofeng(a)cn.fujitsu.com&gt; wrote: >>>>>>> On 08/21/2013 03:06 PM, Eric W. Biederman wrote: >>>>>> >>>>>>>> I suspect libvirt should simply not share /run or any other normally >>>>>>>> writable directory with the host. Sharing /run /var/run or even /tmp >>>>>>>> seems extremely dubious if you want some kind of containment, and >>>>>>>> without strange things spilling through. >>>>>> >>>>>> Right, /run or /var cannot be shared. It's not only about sockets, >>>>>> many other things will also go really wrong that way. >>>>> >>>>> This is very narrow thinking about what a container might be and will >>>>> cause trouble as people start to create novel uses for containers in the >>>>> cloud if you try to impose this on our current infrastructure. >>>>> >>>>> One of the cgroup only container uses we see at Parallels (so no >>>>> separate filesystem and no net namespaces) is pure apache load balancer >>>>> type shared hosting. In this scenario, base apache is effectively >>>>> brought up in the host environment, but then spawned instances are >>>>> resource limited using cgroups according to what the customer has paid. >>>>> Obviously all apache instances are sharing /var and /run from the host >>>>> (mostly for logging and pid storage and static pages). The reason some >>>>> hosters do this is that it allows much higher density simple web serving >>>>> (either static pages from quota limited chroots or dynamic pages limited >>>>> by database space constraints) because each "instance" shares so much >>>>> from the host. The service is obviously much more basic than giving >>>>> each customer a container running apache, but it's much easier for the >>>>> hoster to administer and it serves the customer just as well for a large >>>>> cross section of use cases and for those it doesn't serve, the hoster >>>>> usually has separate container hosting (for a higher price, of course). >>>> >>>> The "container" as we talk about has it's own init, and no, it cannot >>>> share /var or /run. >>> >>> This is what we would call an IaaS container: bringing up init and >>> effectively a new OS inside a container is the closest containers come >>> to being like hypervisors. It's the most common use case of Parallels >>> containers in the field, so I'm certainly not telling you it's a bad >>> idea. >>> >>>> The stuff you talk about has nothing to do with that, it's not >>>> different from all services or a multi-instantiated service on the >>>> host sharing the same /run and /var. >>> >>> I gave you one example: a really simplistic one. A more sophisticated >>> example is a PaaS or SaaS container where you bring the OS up in the >>> host but spawn a particular application into its own container (this is >>> essentially similar to what Docker does). Often in this case, you do >>> add separate mount and network namespaces to make the application >>> isolated and migrateable with its own IP address. The reason you share >>> init and most of the OS from the host is for elasticity and density, >>> which are fast becoming a holy grail type quest of cloud orchestration >>> systems: if you don't have to bring up the OS from init and you can just >>> start the application from a C/R image (orders of magnitude smaller than >>> a full system image) and slap on the necessary namespaces as you clone >>> it, you have something that comes online in miliseconds which is a feat >>> no hypervisor based virtualisation can match. >>> >>> I'm not saying don't pursue the IaaS case, it's definitely useful ... >>> I'm just saying it would be a serious mistake to think that's the only >>> use case for containers and we certainly shouldn't adjust Linux to serve >>> only that use case. >>> >> >> The feature you said above VS contianer-reboot-host bug, I prefer to >> fix >> the bug. > > What bug? > >> and this feature can be achieved even container unshares /run >> directory >> with host by default, for libvirt, user can set the container >> configuration to >> make the container shares the /run directory with host. >> >> I would like to say, the reboot from container bug is more urgent and >> need >> to be fixed. > > Are you talking about the old bug where trying to reboot an lxc > container from within it would reboot the entire system? Yes, we are discussing this problem in this whole thread. If so, OpenVZ > has never suffered from that problem and I thought it was fixed > upstream. I've not tested lxc tools, but the latest vzctl from the > openvz website will bring up a container on the vanilla 3.9 kernel > (provided you have USER_NS compiled in) can also be used to reboot the > container, so I see no reason it wouldn't work for lxc as well. > I'm using libvirt lxc not lxc-tools. Not all of users enable user namespace, I trust these container management tools can have right/proper setting which inhibit this reboot-problem occur. but I don't think this reboot-problem won't happen in any configuration.

Gao feng <gaofeng(a)cn.fujitsu.com&gt; writes: > right now I only take note of the unix socket /run/systemd/private, > but there may have many similar unix sockets, they can exist in any > path. the strange problems will still happen. It could just as easily have been a fifo in the filesystem, and the result would have been the same. The network namespace are all about communicating between network namespaces and that is what was allowed here. If you don't want a socket or a fifo or any other file to be used by a container don't give it access to it. It really is that simple.