Quoting Balbir Singh (balbir(a)linux.vnet.ibm.com): Thanks, Balbir. By the last sentence, are you talking about having cgroup in its own libcgroup, or do you mean something else? On the topic of cgroups, does anyone not agree that we should try to get rid of the ns cgroup, at least once user namespaces can prevent root in a container from escaping their cgroup? thanks, -serge

Serge E. Hallyn wrote: Hi Serge, here are a few suggestions for the containers in general and most of these suggestions are pre-requisites for CR (may be not the higher priority but just to keep in mind). * time virtualization : for absolute timer CR, TCP socket timestamps, ... * inode virtualization : without this you won't be able to migrate some applications eg. samba which rely on the inode numbers. * debugging tools for the containers: at present we are not able to debug a multi-threaded application from outside of the container. * poweroff / reboot from inside the container : at poweroff / reboot, all the processes are killed expect the init process which will stay there making the container blocked. Maybe we can send a SIGINFO signal to the init's parent with some information, so it will be up the parent to: - ignore the signal - stop the container (poweroff/halt) - stop and start again the container (reboot). Right. I know you are working hard on a CR patchset and most of the questions / suggestions below were already addressed in the mailing list since some month ago but IMO they were eluded :) If you can talk about these points and clarify what approach would be preferable that would be nice. IMHO the all-in-kernel-monolithic approach raise some problems: * the tasks are checkpointed from an external process and most of the kernel code is designed to run as current * if a checkpoint or a restart fails, how do we debug that ? How someone in the community using the CR can report an information about the checkpoint has failed in a particular place ? The same for the restart. And a much more harder case is if a restart succeeded but a resource was badly restored making the application to continue its execution but failing 1 hour later. * how this can be maintained ? who will port the CR each time a subsystem design changes ? * the current patchset is full kernel but needs an external tool to create the process tree by digging in the statefile, weird. * the container and the checkpoint/restart are not clearly decorrelated, that brings a dangerous heuristic in the kernel, especially with nested namespace and partial resources checkpoint. IMHO, the checkpoint / restart should succeed even if the resources are not isolated, we should not CR some boundaries like the namespaces. Regarding these points and the comments of Kerrighed and google guys, maybe it would be interesting to discuss the following design of the CR: 1) create a synchronism barrier (not the freezer), where all the tasks can set the checkpoint or restart status That allows to have a task to abort the checkpoint at any time by setting a status error in the synchronism barrier. The initiator of the checkpoint / restart is blocked on this barrier until the checkpoint / restart finishes or fails. If the initiator exits, that's cancel the current operation making possible to do Ctrl+C at checkpoint or restart time. 2) make a vdso which is the entry point of the checkpoint and set this entry as a signal handler for a new signal SIGCKPT, the same for SIGRESTART (AFAIR this is defined in posix 1003.m). This approach allows to checkpoint from the current context which is less arch dependant and/or to override the handler with a specific library making possible to do some work before calling the sys_checkpoint itself. That will allows to build the CR step by step by making in userspace a best-effort library to checkpoint/restart what is not supported in the kernel. 3) a process gains the checkpointable property with a specific flag or whatever. All the childs inherit this flag. That will allows to identify all the tasks which are checkpointable without isolating anything and than opens the door to the checkpoint/restart of a subset of a process tree. 4) dump everything in a core-file-like and improve the interpreter to recreate the process tree from this file. Dynamic behaviour would be: Checkpoint: - The initiator of the checkpoint initialize the barrier and send a signal SIGCKPT to all the checkpointable tasks and these ones will jump on the handler and block on the barrier. - When all these tasks reach this barrier, the initiator of the checkpoint dumps the system wide resources (memory, sysv ipc, struct files, etc ...). - When this is done, the tasks are released and they store their process wide resources (semundo, file descriptor, etc ...) to a current->ckpt_restart buffer and then set the status of the operation and block on the barrier. - The initiator of the checkpoint then collects all these informations and dump them. - Finally the initiator of the checkpoint release the tasks. Restart: - The user executes the statefile, that spawns the process tree and all the processes are blocked in the barrier. - The initiator of the restart restore the system wide resources and fill the restarted processes' current->ckpt_restart buffer. - The initiator sends a SIGRESTART to all the tasks and unblock the tasks - all the tasks restore their process wide resources regarding the current->ckpt_restart buffer. - all the tasks write their status and block on the barrier - the initiator of the restart release the tasks which will return to their execution context when they were checkpointed. This approach is different of you are doing but I am pretty sure most of the code is re-usable. I see different advantages of this approach: - because the process resources are checkpointed / restarted from current, it would be easy to reuse some syscalls code (from the kernel POV) and that would reduce the code duplication and maintenance overhead. - the approach is more fine grained as we can implement piece by piece the checkpoint / restart. - as the statefile is in the elf format, gdb could be used to debug a statefile as a core file - as each process checkpoint / restart themselves, most of the execution context is stored in the stack which is CR with the memory, so when returning from the signal handler, the process returns to the right context. That is less complicated and more generic than externally checkpoint the execution context of a frozen task which would be potentially different for the restart. I hope Serge you can present this approach as an alternative of the current patchset __if__ this one is not acceptable. Regards -- Daniel

Hi Daniel, This is a fair-sized list of issues ... must have been cooking for a while ? ... Daniel Lezcano wrote: Good point. Hmmm... have you given it a thought ? Why not ? does ptrace-ing from parent container not work ? Hmmm... anouther round ? :( So, clearly, I couldn't restist :p I think we are already mostly reusing codes, with few exceptions. Can you elaborate where's the problem ? For checkpoint we have a nice mechanism that adds (a) record(s) to the checkpoint image that describe the error when it occurs. There are a few examples already in the code. We haven't made much progress on the restart front, yet. I'm pretty sure any idea to this end is applicable in either approach. It uses the head of the data to create the process hierarchy. What's weird about it ? The main advantage is the flexibility it provides. The alternative is to start all tasks in the kernel (a la OpenVZ), or what you suggest, which sounds like .. hmm .. external tool to create the process tree by digging in the statefile :p That's already possible in the current approach. This is already how it works in restart. ^^^^^^^^^^^ Is this an issue with current approach ? BTW, to be able to checkpoint at _any time_, preemptively, you _must_ be able to checkpoint externally to the tasks. For instance, how would you handle a ptraced task ? STOPed task ? Aborting using ctrl-c or any other method is already possible now with no harm done. In fact, with less harm than when requiring the cooperation of participating tasks. Why is it less arch dependent ? The only arch dependent code in the current patchset is what is defined differently by separate archs (cpus, mm-context). This sort of notification is indeed desirable and can be added to either approach. Already possible. Isolation is a nice feature, not a requirement (at least if you ask me :) How is this different from above ? Note that with namespaces, there are no "system wide resources", but instead there are multiple namespaces with resources. Can you explain why this approach is better than the current one ? Rename "initiator" to "external checkpointer", and all the rest is nearly the same. Only that instead of relying on the freezer code (which is, clearly, reuse of existing code!), your approach requires a delicate mechanism to allow all tasks to cooperate at the initiator's will. Done already. Done already (with the exception that they do it one by one because the checkpoint image is streamed). Done. Ditto. Checkpoint and restart are asymmetric: checkpoint needs to _observe_ and record, and restart needs to _create_ and build. That's why reusing existing syscalls is extremely helpful for restart, but not so much for checkpoint. In current approach, restart indeed is done in the current context. And that's where you'd like to reuse syscalls. Checkpoint is done by observing tasks (out of their context), and I believe the code will be pretty much the same for in-context. Being out of context requires little bit glue to guarantee safe access to non-current resources. Can do. Was discussed on containers mailing list some time ago with Kerrighead, IIRC in regarding IPC namespaces. Ehh ? The code is actually straight forward. No kernel stack, and user stack is in memory anyway. Take a look at the code, it's pretty straightforward. There you go. I could not resist :O Now, before I go hide (...) - some of these points require attention, e.g. - error reporting on restart, notification mechanisms, partial containers and selected resources, etc. Oren.

Quoting Oren Laadan (orenl(a)cs.columbia.edu): Yup, these are all things we need to discuss. For some of them we might just need to flail about and code a few approaches until we figure out an answer, but then I think that everyone has thought about a few of these in some detail, so there probably is much we could gain from talking. ... Does this mean we should try to have a mini-summit in the next 6 months or so? I'd recommend having one right before kernel summit so we can get our act together, but getting everyone to tokyo to chat seems uneconomical :) It'd be good to chat about at least the first two items before the summit, though. Maybe after we finish v17, we pick a few of these and try a focused push to get answers?

Serge E. Hallyn wrote: How about linux plumbers ? Oren.

Serge E. Hallyn wrote: Seems like I can make it. Oren.

Quoting Daniel Lezcano (dlezcano(a)fr.ibm.com): Absolutely. Of course it's not until October. I'll be sending out a copy of the notes I take with me (including the info from this thread) beforehand. thanks, -serge