6:58 p.m.
Dear all,
I have a thirty-party PCIe card in my host which can work properly in native mode.
I wonder if this card can be shared by multiple containers.
As far as the network interface is concerned, too many efforts have been made for it. Two
dedicated cgroup resource schedulers/subsystem(net_cls, net_prio) have been provided at
the kernel level, and libvirt has also done a lot of work on virtual networking to enable
network interface sharing between containers.
As it's difficult for me to implement all these things that have been done for network
interface, I should find other way to enable sharing of this card.
Do you have any idea?
Any comments will be high appreciated.
Cheng Wang
4:45 a.m.
On Wed, Jan 15, 2014 at 12:58:54AM +0000, WANG Cheng D wrote:
Dear all,
I have a thirty-party PCIe card in my host which can work properly in native mode.
I wonder if this card can be shared by multiple containers.
As far as the network interface is concerned, too many efforts have
been made for it. Two dedicated cgroup resource schedulers/subsystem
(net_cls, net_prio) have been provided at the kernel level, and
libvirt has also done a lot of work on virtual networking to enable
network interface sharing between containers.
As it's difficult for me to implement all these things that have been
done for network interface, I should find other way to enable sharing
of this card.
Do you have any idea?
You don't mention what kind of PCI card it is and how you want the
guest to use it. It may or may not be possible to share it depending
on the answer to those questions.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
2:55 a.m.
Dear Daniel,
The thirty-party PCIe card is based on the Xilinx’ FPGA which is off the shelf, the main
features are as follows:
1) x8 Gen3, 8Gb/s per lane/direction
2) MSI and legacy interrupt support
3) Scatter-gather packet DMA engine provide by Northwest Logic
We hope multiple Linux Containers to access the PCIe card in time division mode, for
example, during slot 1, lxc1 read/write the PCIe card; during slot 2, lxc2 read/write the
PCIe card.
Taking reading operation (data flows from PCIe card to host memory) as example, in native
mode, the PCIe driver should tell PCIe card the host memory address where to store the
incoming data from PCIe card. With this address, the PCIe card will transmit the data to
host memory itself.
In container case, the problem is how application running in the container knows the
expected host memory address.
I wonder if we can have a process running in the which solely operates the PCIe card, all
the containers will send/receive data to this process via socket.
With my warmest regards,
Cheng Wang
-----Original Message-----
From: Daniel P. Berrange [mailto:berrange@redhat.com]
Sent: 2014年1月15日 18:46
To: WANG Cheng D
Cc: libvirt-users(a)redhat.com
Subject: Re: [libvirt-users] If it's possible for a third-party PCIe card to be shared
by multiple containers
On Wed, Jan 15, 2014 at 12:58:54AM +0000, WANG Cheng D wrote:
Dear all,
I have a thirty-party PCIe card in my host which can work properly in native mode.
I wonder if this card can be shared by multiple containers.
As far as the network interface is concerned, too many efforts have
been made for it. Two dedicated cgroup resource schedulers/subsystem
(net_cls, net_prio) have been provided at the kernel level, and
libvirt has also done a lot of work on virtual networking to enable
network interface sharing between containers.
As it's difficult for me to implement all these things that have been
done for network interface, I should find other way to enable sharing
of this card.
Do you have any idea?
You don't mention what kind of PCI card it is and how you want the guest to use it. It
may or may not be possible to share it depending on the answer to those questions.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
4:34 a.m.
On Thu, Jan 16, 2014 at 08:55:53AM +0000, WANG Cheng D wrote:
Dear Daniel,
The thirty-party PCIe card is based on the Xilinx’ FPGA which is off the shelf, the main
features are as follows:
1) x8 Gen3, 8Gb/s per lane/direction
2) MSI and legacy interrupt support
3) Scatter-gather packet DMA engine provide by Northwest Logic
We hope multiple Linux Containers to access the PCIe card in time division mode, for
example, during slot 1, lxc1 read/write the PCIe card; during slot 2, lxc2 read/write the
PCIe card.
Taking reading operation (data flows from PCIe card to host memory) as example, in native
mode, the PCIe driver should tell PCIe card the host memory address where to store the
incoming data from PCIe card. With this address, the PCIe card will transmit the data to
host memory itself.
In container case, the problem is how application running in the container knows the
expected host memory address.
I wonder if we can have a process running in the which solely operates the PCIe card, all
the containers will send/receive data to this process via socket.
Ok, so from the LXC driver POV I guess this card is accessed
via some special device node with ioctls or something ? If so
then you can just expose the device node to as many containers
as you like at the same time without restrictions.
http://libvirt.org/formatdomain.html#elementsHostDevCaps
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
2:37 a.m.
Dear Daniel,
Thank you for the information. I think I should study more about libvirt's Node Device
functions.
Have a nice day.
Cheng
-----Original Message-----
From: Daniel P. Berrange [mailto:berrange@redhat.com]
Sent: 2014年1月16日 18:34
To: WANG Cheng D
Cc: libvirt-users(a)redhat.com
Subject: Re: [libvirt-users] If it's possible for a third-party PCIe card to be shared
by multiple containers
On Thu, Jan 16, 2014 at 08:55:53AM +0000, WANG Cheng D wrote:
Dear Daniel,
The thirty-party PCIe card is based on the Xilinx’ FPGA which is off the shelf, the main
features are as follows:
1) x8 Gen3, 8Gb/s per lane/direction
2) MSI and legacy interrupt support
3) Scatter-gather packet DMA engine provide by Northwest Logic We hope
multiple Linux Containers to access the PCIe card in time division mode, for example,
during slot 1, lxc1 read/write the PCIe card; during slot 2, lxc2 read/write the PCIe
card.
Taking reading operation (data flows from PCIe card to host memory) as example, in native
mode, the PCIe driver should tell PCIe card the host memory address where to store the
incoming data from PCIe card. With this address, the PCIe card will transmit the data to
host memory itself.
In container case, the problem is how application running in the container knows the
expected host memory address.
I wonder if we can have a process running in the which solely operates the PCIe card, all
the containers will send/receive data to this process via socket.
Ok, so from the LXC driver POV I guess this card is accessed via some special device node
with ioctls or something ? If so then you can just expose the device node to as many
containers as you like at the same time without restrictions.
http://libvirt.org/formatdomain.html#elementsHostDevCaps
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
2:54 a.m.
Dear Daniel,
I found that lxc driver doesn't support hostdev with type of 'pci'.
As you mentioned before, I tried to expose the device node (the PCIe card) to the
container, my XML is as follow:
<devices>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address bus='0x01' slot='0x00' function='0x0'/>
</source>
</hostdev>
</devices>
When I started the container, I got the error message:
Error: failed to start domain
Error: unsupported configuration: Unsupported hostdev type pci.
I want to confirm if the element hostdev of PCIe device is not supported by lxc driver
currently.
Have a nice day,
Cheng Wang
-----Original Message-----
From: Daniel P. Berrange [mailto:berrange@redhat.com]
Sent: 2014年1月16日 18:34
To: WANG Cheng D
Cc: libvirt-users(a)redhat.com
Subject: Re: [libvirt-users] If it's possible for a third-party PCIe card to be shared
by multiple containers
On Thu, Jan 16, 2014 at 08:55:53AM +0000, WANG Cheng D wrote:
Dear Daniel,
The thirty-party PCIe card is based on the Xilinx’ FPGA which is off the shelf, the main
features are as follows:
1) x8 Gen3, 8Gb/s per lane/direction
2) MSI and legacy interrupt support
3) Scatter-gather packet DMA engine provide by Northwest Logic We hope
multiple Linux Containers to access the PCIe card in time division mode, for example,
during slot 1, lxc1 read/write the PCIe card; during slot 2, lxc2 read/write the PCIe
card.
Taking reading operation (data flows from PCIe card to host memory) as example, in native
mode, the PCIe driver should tell PCIe card the host memory address where to store the
incoming data from PCIe card. With this address, the PCIe card will transmit the data to
host memory itself.
In container case, the problem is how application running in the container knows the
expected host memory address.
I wonder if we can have a process running in the which solely operates the PCIe card, all
the containers will send/receive data to this process via socket.
Ok, so from the LXC driver POV I guess this card is accessed via some special device node
with ioctls or something ? If so then you can just expose the device node to as many
containers as you like at the same time without restrictions.
http://libvirt.org/formatdomain.html#elementsHostDevCaps
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
4:27 a.m.
On Thu, Jan 23, 2014 at 08:54:21AM +0000, WANG Cheng D wrote:
Dear Daniel,
I found that lxc driver doesn't support hostdev with type of 'pci'.
As you mentioned before, I tried to expose the device node (the PCIe card) to the
container, my XML is as follow:
<devices>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address bus='0x01' slot='0x00' function='0x0'/>
</source>
</hostdev>
</devices>
When I started the container, I got the error message:
Error: failed to start domain
Error: unsupported configuration: Unsupported hostdev type pci.
I want to confirm if the element hostdev of PCIe device is not supported by lxc driver
currently.
The PCI mode is for actual PCI device assignment at the hardware level.
This concept doesn't make sense for containers since there's a shared
kernel. Instead you want to assign the device nodes from /dev using
the mode='capability' hostdev XML instead per this link:
http://libvirt.org/formatdomain.html#elementsHostDevCaps
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
3990
days inactive
3998
days old
6 comments
2 participants
participants (2)
-
Daniel P. Berrange -
WANG Cheng D