Hi Jan, Thanks for the reply. I looked deeper into the libvirt and Jailhouse source code and found following two things that seem relevant to the project I am interested in. - Libvirt driver interface at [libvirt.git] <https://libvirt.org/git/?p=libvirt.git;a=tree;hb=HEAD> / src <https://libvirt.org/git/?p=libvirt.git;a=tree;f=src;hb=HEAD> / driver.h <https://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=src/driver.h;hb=HEA... - Jailhouse tool, which is using the ioctl API of the Jailhouse, available at https://github.com/siemens/jailhouse/blob/master/tools/jailhouse.c. With the help of the above two, it looks like, a libvirt driver for the Jailhouse can be implemented. Let me know if I am moving in the right direction so far. I have been looking at the other libvirt driver implementations for hypervisors like HyperV and VMware to understand their implementation and learn from there. Thanks & Regards, Prakhar Bansal Graduate Student' 20, Computer Engineering Iowa State University, Ames, IA On Mon, Mar 23, 2020 at 3:07 AM Jan Kiszka <jan.kiszka(a)siemens.com&gt; wrote: > Hi Prakhar, > > On 23.03.20 07:53, PRAKHAR BANSAL wrote: > > Hello All, > > > > My name is Prakhar Bansal and I am a graduate student in Computer > > Engineering at Iowa State University, US. > > I have experience with Analysing Performance of Applications running > > inside multiple virtual machines hosted by the libvirt QEMU-KVM through > > virt-manager. > > > > I am interested in working on the project to develop a Libvirt driver > > for the Jailhouse hypervisor. I looked into the initial attempt on the > > Jailhouse driver which seems to be based on the Jailhouse command-line > > interface. I am currently looking into learning and understanding the > > kernel APIs for jailhouse hypervisor. > > Thanks for your interest! > > > I followed the below articles mentioned by Valentine Sinitsyn to begin > > learning about the Jailhouse hypervisor. > > > > https://lwn.net/Articles/578295/ > > https://lwn.net/Articles/578852/ > > > > I have a few questions regarding this project, please let me know if > > someone can help me out. > > Sure, go ahead. Depending on the scope of the question, libvirt might be > the better community to ask. Therefore, I'm adding its list to this > thread. > > Jan > > > > > Thanks & Regards, > > Prakhar Bansal > > > > -- > Siemens AG, Corporate Technology, CT RDA IOT SES-DE > Corporate Competence Center Embedded Linux >

Hi Prakhar, On 25.03.20 05:36, PRAKHAR BANSAL wrote: > Hi Jan, > > Thanks for the reply. I looked deeper into the libvirt and Jailhouse > source code and found following two things that seem relevant to the > project I am interested in. > > - Libvirt driver interface at [libvirt.git] > <https://libvirt.org/git/?p=libvirt.git;a=tree;hb=HEAD> / src > <https://libvirt.org/git/?p=libvirt.git;a=tree;f=src;hb=HEAD> / driver.h > < https://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=src/driver.h;hb=HEAD > > - Jailhouse tool, which is using the ioctl API of the Jailhouse, > available at > https://github.com/siemens/jailhouse/blob/master/tools/jailhouse.c. > > With the help of the above two, it looks like, a libvirt driver for the > Jailhouse can be implemented. Let me know if I am moving in the right > direction so far. From the Jailhouse perspective, it is important to not consider the command line tool an interface anymore (like in the first prototype) but build on top of the Linux driver API (ioctls, sysfs). There is already a Python library which started to abstract this interface for Jailhouse-internal use cases. However, I strongly suspect libvirt will rather want a native binding. > > I have been looking at the other libvirt driver implementations for > hypervisors like HyperV and VMware to understand their implementation > and learn from there. As Jailhouse is a static partitioning hypervisor without abstraction of the underlying hardware, your starting point for the libvirt binding should be a given set of Jailhouse cell configurations describing a complete partitioned system. So rather than instantiating on demand a domain (Jailhouse cell) with, say, a network adapter, the driver should match a user request for a domain against the configuration set and use what is there - or report the mismatch. What it could organize, though, is interconnecting cells that have a (preconfigured) virtual network link to the root cell. Due to this different concept, there will be no 1:1 mapping for commodity hypervisor drivers to the Jailhouse scenario. Still, studying what they do is useful and needed in order to understand what "normally" happens and find a reasonable translation. This is probably the most challenging part of the project. Jan

On 28.03.20 08:47, PRAKHAR BANSAL wrote: > Hi Jan, > > Thanks for the reply! > > I was only considering the command-line tool "code" for reference to the > Jailhouse kernel API(ioctl calls) because I didn't find a documentation > of the Jailhouse kernel APIs. Right, the IOCTL API is not documented so far. It is currently only used inside the Jailhouse project. This needs to be formalized when there shall be external users like a libvirt driver. That might be a nice small contribution task: Create some Documentation/driver-interfaces.md that describes the IOCTLs along with their parameter structures and that also includes the current sysfs-entries.txt as a section. Then send this as patch here. I'll help out when details are not clear from reading the code. Sure. I will do that.

> > For the second part as you mentioned that Jailhouse can only create > cells with the constraints defined in the root cell configuration. I > have a few questions regarding that. > > 1. Is there a way to know if Jailhouse is enabled on the host and get > the root cell configuration(s) from Jailhouse through an API? This can > be used while binding the libvirt to the Jailhouse hypervisor. Look at https://github.com/siemens/jailhouse/blob/master/Documentation/sysfs-entr... for what is reported as runtime information. Full configurations can't be read back at this point. This might be reconsidered in the light of [1], but I wouldn't plat for that yet.

> > 2. If Jailhouse is not enabled(again can we know this using some API) > then, can libvirt enable/disable Jailhouse during the libvirt binding of > the Jailhouse driver with a given set of Jailhouse cell configurations > describing a complete partitioned system? With the API above and a given configuration set, yes. The config set would have to be provided to the libvirt driver in some to-be-defined way (e.g. /etc/libvirt/jailhouse.conf -> /etc/libvirt/jailhouse/*.cell). Cool, got it. Thanks!

> > 3. I was wondering, as you mentioned that libvirt driver should check > for mismatch of the cell configuration with the root cell configuration, > the question is, isn't that done by Jailhouse itself? If yes, then > libvirt can just pass on the cell creation requests to Jailhouse and > return the response to the user as it is, rather than driver doing any > such mismatch check. With matching I'm referring to a libvirt user request like "create a domain with 2 CPUs", while there are no cells left that have more than one CPU. Or "give the domain 1G RAM", and you need to find an available cell with that much memory. Those are simple examples. A request that states "connect the domain to the host network A" implies that a cell has a shared-memory link to, say, the root cell that can be configured to bridge this. But let's keep that for later and start as simple as possible.

Jan [1] https://groups.google.com/d/msgid/jailhouse-dev/CADiTV-1QiRhSWZnw%2BkHhJM... > > -Prakhar > > On Thu, Mar 26, 2020 at 1:49 AM Jan Kiszka <jan.kiszka(a)web.de > <mailto:jan.kiszka@web.de>> wrote: > > Hi Prakhar, > > On 25.03.20 05:36, PRAKHAR BANSAL wrote: > > Hi Jan, > > > > Thanks for the reply. I looked deeper into the libvirt and Jailhouse > > source code and found following two things that seem relevant to the > > project I am interested in. > > > > - Libvirt driver interface at [libvirt.git] > > <https://libvirt.org/git/?p=libvirt.git;a=tree;hb=HEAD> / src > > <https://libvirt.org/git/?p=libvirt.git;a=tree;f=src;hb=HEAD> / > driver.h > > > < https://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=src/driver.h;hb=HEAD > > > - Jailhouse tool, which is using the ioctl API of the Jailhouse, > > available at > > https://github.com/siemens/jailhouse/blob/master/tools/jailhouse.c. > > > > With the help of the above two, it looks like, a libvirt driver > for the > > Jailhouse can be implemented. Let me know if I am moving in the right > > direction so far. > > From the Jailhouse perspective, it is important to not consider the > command line tool an interface anymore (like in the first prototype) but > build on top of the Linux driver API (ioctls, sysfs). There is already a > Python library which started to abstract this interface for > Jailhouse-internal use cases. However, I strongly suspect libvirt will > rather want a native binding. > > > > > I have been looking at the other libvirt driver implementations for > > hypervisors like HyperV and VMware to understand their implementation > > and learn from there. > > As Jailhouse is a static partitioning hypervisor without abstraction of > the underlying hardware, your starting point for the libvirt binding > should be a given set of Jailhouse cell configurations describing a > complete partitioned system. So rather than instantiating on demand a > domain (Jailhouse cell) with, say, a network adapter, the driver should > match a user request for a domain against the configuration set and use > what is there - or report the mismatch. What it could organize, though, > is interconnecting cells that have a (preconfigured) virtual network > link to the root cell. > > Due to this different concept, there will be no 1:1 mapping for > commodity hypervisor drivers to the Jailhouse scenario. Still, studying > what they do is useful and needed in order to understand what "normally" > happens and find a reasonable translation. This is probably the most > challenging part of the project. > > Jan >

On 30.03.20 10:02, PRAKHAR BANSAL wrote: > Hi Jan, > > On Sat, Mar 28, 2020 at 4:12 AM Jan Kiszka <jan.kiszka(a)web.de > <mailto:jan.kiszka@web.de>> wrote: > > On 28.03.20 08:47, PRAKHAR BANSAL wrote: > > Hi Jan, > > > > Thanks for the reply! > > > > I was only considering the command-line tool "code" for reference > to the > > Jailhouse kernel API(ioctl calls) because I didn't find a > documentation > > of the Jailhouse kernel APIs. > > Right, the IOCTL API is not documented so far. It is currently only used > inside the Jailhouse project. This needs to be formalized when there > shall be external users like a libvirt driver. > > That might be a nice small contribution task: Create some > Documentation/driver-interfaces.md that describes the IOCTLs along with > their parameter structures and that also includes the current > sysfs-entries.txt as a section. Then send this as patch here. I'll help > out when details are not clear from reading the code. > > Sure. I will do that. > > > > > For the second part as you mentioned that Jailhouse can only create > > cells with the constraints defined in the root cell configuration. I > > have a few questions regarding that. > > > > 1. Is there a way to know if Jailhouse is enabled on the host and get > > the root cell configuration(s) from Jailhouse through an API? > This can > > be used while binding the libvirt to the Jailhouse hypervisor. > > Look at > https://github.com/siemens/jailhouse/blob/master/Documentation/sysfs-entr... > for what is reported as runtime information. Full configurations can't > be read back at this point. This might be reconsidered in the light of > [1], but I wouldn't plat for that yet. > > > Ok, sure. I am looking into it. > > > > > > 2. If Jailhouse is not enabled(again can we know this using some API) > > then, can libvirt enable/disable Jailhouse during the libvirt > binding of > > the Jailhouse driver with a given set of Jailhouse cell > configurations > > describing a complete partitioned system? > > With the API above and a given configuration set, yes. The config set > would have to be provided to the libvirt driver in some to-be-defined > way (e.g. /etc/libvirt/jailhouse.conf -> /etc/libvirt/jailhouse/*.cell). > > Cool, got it. Thanks! > > > > > 3. I was wondering, as you mentioned that libvirt driver should check > > for mismatch of the cell configuration with the root cell > configuration, > > the question is, isn't that done by Jailhouse itself? If yes, then > > libvirt can just pass on the cell creation requests to Jailhouse and > > return the response to the user as it is, rather than driver > doing any > > such mismatch check. > > With matching I'm referring to a libvirt user request like "create a > domain with 2 CPUs", while there are no cells left that have more than > one CPU. Or "give the domain 1G RAM", and you need to find an available > cell with that much memory. Those are simple examples. A request that > states "connect the domain to the host network A" implies that a cell > has a shared-memory link to, say, the root cell that can be configured > to bridge this. But let's keep that for later and start as simple as > possible. > > > Do I need to match the libvirt user-requested cell config with only root > cells or with all cells present at that time? With all non-root cells - the root cell will be occupied already (it runs libvirt e.g.). > > I wanted to request you for a favor for the proposal as the deadline is > approaching. Could I prepare a proposal for this project based on our > discussion here and improve it later based on feedback comments after > the deadline? I understand that I got late in starting on the project > search and selection. Sure, please go ahead. Jan > > Thanks, > Prakhar > > > Jan > > [1] > https://groups.google.com/d/msgid/jailhouse-dev/CADiTV-1QiRhSWZnw%2BkHhJM... > > > > > -Prakhar > > > > On Thu, Mar 26, 2020 at 1:49 AM Jan Kiszka <jan.kiszka(a)web.de > <mailto:jan.kiszka@web.de> > > <mailto:jan.kiszka@web.de <mailto:jan.kiszka@web.de>>> wrote: > > > > Hi Prakhar, > > > > On 25.03.20 05:36, PRAKHAR BANSAL wrote: > > > Hi Jan, > > > > > > Thanks for the reply. I looked deeper into the libvirt and > Jailhouse > > > source code and found following two things that seem > relevant to the > > > project I am interested in. > > > > > > - Libvirt driver interface at [libvirt.git] > > > <https://libvirt.org/git/?p=libvirt.git;a=tree;hb=HEAD> / src > > > > <https://libvirt.org/git/?p=libvirt.git;a=tree;f=src;hb=HEAD> / > > driver.h > > > > > > < https://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=src/driver.h;hb=HEAD > > > > - Jailhouse tool, which is using the ioctl API of the > Jailhouse, > > > available at > > > > https://github.com/siemens/jailhouse/blob/master/tools/jailhouse.c. > > > > > > With the help of the above two, it looks like, a libvirt > driver > > for the > > > Jailhouse can be implemented. Let me know if I am moving > in the right > > > direction so far. > > > > From the Jailhouse perspective, it is important to not > consider the > > command line tool an interface anymore (like in the first > prototype) but > > build on top of the Linux driver API (ioctls, sysfs). There > is already a > > Python library which started to abstract this interface for > > Jailhouse-internal use cases. However, I strongly suspect > libvirt will > > rather want a native binding. > > > > > > > > I have been looking at the other libvirt driver > implementations for > > > hypervisors like HyperV and VMware to understand their > implementation > > > and learn from there. > > > > As Jailhouse is a static partitioning hypervisor without > abstraction of > > the underlying hardware, your starting point for the libvirt > binding > > should be a given set of Jailhouse cell configurations > describing a > > complete partitioned system. So rather than instantiating on > demand a > > domain (Jailhouse cell) with, say, a network adapter, the > driver should > > match a user request for a domain against the configuration > set and use > > what is there - or report the mismatch. What it could > organize, though, > > is interconnecting cells that have a (preconfigured) virtual > network > > link to the root cell. > > > > Due to this different concept, there will be no 1:1 mapping for > > commodity hypervisor drivers to the Jailhouse scenario. > Still, studying > > what they do is useful and needed in order to understand what > "normally" > > happens and find a reasonable translation. This is probably > the most > > challenging part of the project. > > > > Jan > > >

On 31.03.20 04:43, PRAKHAR BANSAL wrote: > Hi Jan, > > Thanks for the confirmation to proceed on project proposal. > > Also, I tried installing Jailhouse on my VM after enabling VT-x/EPT and > IOMMU for my VM(Guest OS- Ubuntu 18.04) on VMware fusion hypervisor with > MacOS on the host side. > However, Jailhouse hardware check was failed because of missing > *Preemption timer and Virtualize APIC access*, could you please suggest, > if this is hardware limitation? Is there any workaround here? You will need a hypervisor that supports both when nesting, but I have no idea if there is one for the Mac. What is known to work is KVM on Linux hosts. > My laptop's processor is Intel quad-core i7. > > image.png > > Also, could you please suggest, if I can talk to you through an IRC or > slack channel since it is a bit hard to communicate over email every time. I'll be listening on #jailhouse, irc.freenode.net. Jan > > Thanks, > Prakhar > > > On Mon, Mar 30, 2020 at 6:15 AM Jan Kiszka <jan.kiszka(a)web.de > <mailto:jan.kiszka@web.de>> wrote: > > On 30.03.20 10:02, PRAKHAR BANSAL wrote: > > Hi Jan, > > > > On Sat, Mar 28, 2020 at 4:12 AM Jan Kiszka <jan.kiszka(a)web.de > <mailto:jan.kiszka@web.de> > > <mailto:jan.kiszka@web.de <mailto:jan.kiszka@web.de>>> wrote: > > > > On 28.03.20 08:47, PRAKHAR BANSAL wrote: > > > Hi Jan, > > > > > > Thanks for the reply! > > > > > > I was only considering the command-line tool "code" for > reference > > to the > > > Jailhouse kernel API(ioctl calls) because I didn't find a > > documentation > > > of the Jailhouse kernel APIs. > > > > Right, the IOCTL API is not documented so far. It is > currently only used > > inside the Jailhouse project. This needs to be formalized > when there > > shall be external users like a libvirt driver. > > > > That might be a nice small contribution task: Create some > > Documentation/driver-interfaces.md that describes the IOCTLs > along with > > their parameter structures and that also includes the current > > sysfs-entries.txt as a section. Then send this as patch here. > I'll help > > out when details are not clear from reading the code. > > > > Sure. I will do that. > > > > > > > > For the second part as you mentioned that Jailhouse can > only create > > > cells with the constraints defined in the root cell > configuration. I > > > have a few questions regarding that. > > > > > > 1. Is there a way to know if Jailhouse is enabled on the > host and get > > > the root cell configuration(s) from Jailhouse through an API? > > This can > > > be used while binding the libvirt to the Jailhouse hypervisor. > > > > Look at > > > https://github.com/siemens/jailhouse/blob/master/Documentation/sysfs-entr... > > for what is reported as runtime information. Full > configurations can't > > be read back at this point. This might be reconsidered in the > light of > > [1], but I wouldn't plat for that yet. > > > > > > Ok, sure. I am looking into it. > > > > > > > > > > 2. If Jailhouse is not enabled(again can we know this > using some API) > > > then, can libvirt enable/disable Jailhouse during the libvirt > > binding of > > > the Jailhouse driver with a given set of Jailhouse cell > > configurations > > > describing a complete partitioned system? > > > > With the API above and a given configuration set, yes. The > config set > > would have to be provided to the libvirt driver in some > to-be-defined > > way (e.g. /etc/libvirt/jailhouse.conf -> > /etc/libvirt/jailhouse/*.cell). > > > > Cool, got it. Thanks! > > > > > > > > 3. I was wondering, as you mentioned that libvirt driver > should check > > > for mismatch of the cell configuration with the root cell > > configuration, > > > the question is, isn't that done by Jailhouse itself? If > yes, then > > > libvirt can just pass on the cell creation requests to > Jailhouse and > > > return the response to the user as it is, rather than driver > > doing any > > > such mismatch check. > > > > With matching I'm referring to a libvirt user request like > "create a > > domain with 2 CPUs", while there are no cells left that have > more than > > one CPU. Or "give the domain 1G RAM", and you need to find an > available > > cell with that much memory. Those are simple examples. A > request that > > states "connect the domain to the host network A" implies > that a cell > > has a shared-memory link to, say, the root cell that can be > configured > > to bridge this. But let's keep that for later and start as > simple as > > possible. > > > > > > Do I need to match the libvirt user-requested cell config with > only root > > cells or with all cells present at that time? > > With all non-root cells - the root cell will be occupied already (it > runs libvirt e.g.). > > > > > I wanted to request you for a favor for the proposal as the > deadline is > > approaching. Could I prepare a proposal for this project based on our > > discussion here and improve it later based on feedback comments after > > the deadline? I understand that I got late in starting on the project > > search and selection. > > Sure, please go ahead. > > Jan > > > > > Thanks, > > Prakhar > > > > > > Jan > > > > [1] > > > https://groups.google.com/d/msgid/jailhouse-dev/CADiTV-1QiRhSWZnw%2BkHhJM... > > > > > > > > -Prakhar > > > > > > On Thu, Mar 26, 2020 at 1:49 AM Jan Kiszka > <jan.kiszka(a)web.de <mailto:jan.kiszka@web.de> > > <mailto:jan.kiszka@web.de <mailto:jan.kiszka@web.de>> > > > <mailto:jan.kiszka@web.de <mailto:jan.kiszka@web.de> > <mailto:jan.kiszka@web.de <mailto:jan.kiszka@web.de>>>> wrote: > > > > > > Hi Prakhar, > > > > > > On 25.03.20 05:36, PRAKHAR BANSAL wrote: > > > > Hi Jan, > > > > > > > > Thanks for the reply. I looked deeper into the > libvirt and > > Jailhouse > > > > source code and found following two things that seem > > relevant to the > > > > project I am interested in. > > > > > > > > - Libvirt driver interface at [libvirt.git] > > > > > <https://libvirt.org/git/?p=libvirt.git;a=tree;hb=HEAD> / src > > > > > > <https://libvirt.org/git/?p=libvirt.git;a=tree;f=src;hb=HEAD > / > > > driver.h > > > > > > > > > > < https://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=src/driver.h;hb=HEAD > > > > > - Jailhouse tool, which is using the ioctl API of the > > Jailhouse, > > > > available at > > > > > > https://github.com/siemens/jailhouse/blob/master/tools/jailhouse.c. > > > > > > > > With the help of the above two, it looks like, a > libvirt > > driver > > > for the > > > > Jailhouse can be implemented. Let me know if I am > moving > > in the right > > > > direction so far. > > > > > > From the Jailhouse perspective, it is important to not > > consider the > > > command line tool an interface anymore (like in the first > > prototype) but > > > build on top of the Linux driver API (ioctls, sysfs). > There > > is already a > > > Python library which started to abstract this > interface for > > > Jailhouse-internal use cases. However, I strongly suspect > > libvirt will > > > rather want a native binding. > > > > > > > > > > > I have been looking at the other libvirt driver > > implementations for > > > > hypervisors like HyperV and VMware to understand their > > implementation > > > > and learn from there. > > > > > > As Jailhouse is a static partitioning hypervisor without > > abstraction of > > > the underlying hardware, your starting point for the > libvirt > > binding > > > should be a given set of Jailhouse cell configurations > > describing a > > > complete partitioned system. So rather than > instantiating on > > demand a > > > domain (Jailhouse cell) with, say, a network adapter, the > > driver should > > > match a user request for a domain against the > configuration > > set and use > > > what is there - or report the mismatch. What it could > > organize, though, > > > is interconnecting cells that have a (preconfigured) > virtual > > network > > > link to the root cell. > > > > > > Due to this different concept, there will be no 1:1 > mapping for > > > commodity hypervisor drivers to the Jailhouse scenario. > > Still, studying > > > what they do is useful and needed in order to > understand what > > "normally" > > > happens and find a reasonable translation. This is > probably > > the most > > > challenging part of the project. > > > > > > Jan > > > > > >