[libvirt-users] method for communication between containers
Dear all, In my system, two containers need to exchange data as quick as possible and the two containers are hosted by the same physical machine, I wonder if socket is the only method for communication between containers? Thank you. Cheng Wang
On Thu, Feb 27, 2014 at 06:28:46AM +0000, WANG Cheng D wrote:
Dear all, In my system, two containers need to exchange data as quick as possible and the two containers are hosted by the same physical machine, I wonder if socket is the only method for communication between containers?
If you have an area of filesystem that is shared + writable to both containers, they could create UNIX sockets in that filesystem, rather than TCP sockets. Also they could create named FIFO pipes. Another option would be to create a shared memory backed file in shmfs or hugepagefs and expose it to both containers. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Dear Daniel, Thank you for your email. According to your suggestion, I tried to use the shared memory filesystem for data exchange between the host machine and the container. The steps are as follows: 1) create an empty file in the host machine (/tmp1/file1). 2) run #chmod 777 /tmp1 and #chmod 777 /tmp1/file1. 3) expose /tmp1 to the container by configuring the container with the following XML scripts. <filesystem type="mount" accessmode="passthrough"> <source dir="/tmp1"/> <target dir="/tmp1"/> </filesystem> 4) write two programs: program1 writes a string to the shared memory program2 reads the string from the shared memory and compile both programs in the host machine. both of the two programs use ftok() and shmget() to obtain the shmfs id. 5) a. if we run both program1 and program2 in the host machine, the program2 can read the string that program1 has written. b. if we run both program1 and program2 in the container, both programs also work well. c. if we run program1 in the host and run program2 in the container, program2 fails. the ftok() function can return the expected value, but shmget(...,0,0) fails and prompts "No such files". It seems that the exposed file cannot be recognized by shmget() function, although ftok() can work and return the value. Do you have an idea? Best wishes. Cheng Wang -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: 2014年2月27日 18:30 To: WANG Cheng D Cc: libvirt-users@redhat.com Subject: Re: [libvirt-users] method for communication between containers On Thu, Feb 27, 2014 at 06:28:46AM +0000, WANG Cheng D wrote:
Dear all, In my system, two containers need to exchange data as quick as possible and the two containers are hosted by the same physical machine, I wonder if socket is the only method for communication between containers?
If you have an area of filesystem that is shared + writable to both containers, they could create UNIX sockets in that filesystem, rather than TCP sockets. Also they could create named FIFO pipes. Another option would be to create a shared memory backed file in shmfs or hugepagefs and expose it to both containers. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Mar 03, 2014 at 01:39:03PM +0000, WANG Cheng D wrote:
Dear Daniel, Thank you for your email. According to your suggestion, I tried to use the shared memory filesystem for data exchange between the host machine and the container. The steps are as follows:
1) create an empty file in the host machine (/tmp1/file1). 2) run #chmod 777 /tmp1 and #chmod 777 /tmp1/file1. 3) expose /tmp1 to the container by configuring the container with the following XML scripts. <filesystem type="mount" accessmode="passthrough"> <source dir="/tmp1"/> <target dir="/tmp1"/> </filesystem> 4) write two programs: program1 writes a string to the shared memory program2 reads the string from the shared memory and compile both programs in the host machine. both of the two programs use ftok() and shmget() to obtain the shmfs id. 5) a. if we run both program1 and program2 in the host machine, the program2 can read the string that program1 has written. b. if we run both program1 and program2 in the container, both programs also work well. c. if we run program1 in the host and run program2 in the container, program2 fails. the ftok() function can return the expected value, but shmget(...,0,0) fails and prompts "No such files". It seems that the exposed file cannot be recognized by shmget() function, although ftok() can work and return the value.
Do you have an idea?
Yes, the IPCS namespace causes the container's 'shmget' call to be isolated from the host, so using shmget & other similar SysV-IPC APIs will fail. What I was thinking though was that you'd be able to open() the file in the shared memory fs, and then use mmap() to map its data into RAM in the container. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Dear Daniel, Your suggestion is right. I followed your method and used mmap(), it works well. Now I can realize data exchange between containers. Thank you for your help. Cheng -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: 2014年3月3日 21:42 To: WANG Cheng D Cc: libvirt-users@redhat.com Subject: Re: [libvirt-users] method for communication between containers On Mon, Mar 03, 2014 at 01:39:03PM +0000, WANG Cheng D wrote:
Dear Daniel, Thank you for your email. According to your suggestion, I tried to use the shared memory filesystem for data exchange between the host machine and the container. The steps are as follows:
1) create an empty file in the host machine (/tmp1/file1). 2) run #chmod 777 /tmp1 and #chmod 777 /tmp1/file1. 3) expose /tmp1 to the container by configuring the container with the following XML scripts. <filesystem type="mount" accessmode="passthrough"> <source dir="/tmp1"/> <target dir="/tmp1"/> </filesystem> 4) write two programs: program1 writes a string to the shared memory program2 reads the string from the shared memory and compile both programs in the host machine. both of the two programs use ftok() and shmget() to obtain the shmfs id. 5) a. if we run both program1 and program2 in the host machine, the program2 can read the string that program1 has written. b. if we run both program1 and program2 in the container, both programs also work well. c. if we run program1 in the host and run program2 in the container, program2 fails. the ftok() function can return the expected value, but shmget(...,0,0) fails and prompts "No such files". It seems that the exposed file cannot be recognized by shmget() function, although ftok() can work and return the value.
Do you have an idea?
Yes, the IPCS namespace causes the container's 'shmget' call to be isolated from the host, so using shmget & other similar SysV-IPC APIs will fail. What I was thinking though was that you'd be able to open() the file in the shared memory fs, and then use mmap() to map its data into RAM in the container. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Dear Daniel, As you know, following your suggestion, I can expose shared memory to two containers hosted by the same machine, thus the two containers can exchange data. Now I have a new problem: how does the containers avoid concurrent accesses to the shared memory? In a native system, this can be achieved by using semaphore. But I don't know how the two containers can see the same semaphore. Do I need to design a mechanism to control the access to the shared memory? With my warmest regards, Cheng -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: 2014年3月3日 21:42 To: WANG Cheng D Cc: libvirt-users@redhat.com Subject: Re: [libvirt-users] method for communication between containers On Mon, Mar 03, 2014 at 01:39:03PM +0000, WANG Cheng D wrote:
Dear Daniel, Thank you for your email. According to your suggestion, I tried to use the shared memory filesystem for data exchange between the host machine and the container. The steps are as follows:
1) create an empty file in the host machine (/tmp1/file1). 2) run #chmod 777 /tmp1 and #chmod 777 /tmp1/file1. 3) expose /tmp1 to the container by configuring the container with the following XML scripts. <filesystem type="mount" accessmode="passthrough"> <source dir="/tmp1"/> <target dir="/tmp1"/> </filesystem> 4) write two programs: program1 writes a string to the shared memory program2 reads the string from the shared memory and compile both programs in the host machine. both of the two programs use ftok() and shmget() to obtain the shmfs id. 5) a. if we run both program1 and program2 in the host machine, the program2 can read the string that program1 has written. b. if we run both program1 and program2 in the container, both programs also work well. c. if we run program1 in the host and run program2 in the container, program2 fails. the ftok() function can return the expected value, but shmget(...,0,0) fails and prompts "No such files". It seems that the exposed file cannot be recognized by shmget() function, although ftok() can work and return the value.
Do you have an idea?
Yes, the IPCS namespace causes the container's 'shmget' call to be isolated from the host, so using shmget & other similar SysV-IPC APIs will fail. What I was thinking though was that you'd be able to open() the file in the shared memory fs, and then use mmap() to map its data into RAM in the container. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, Apr 09, 2014 at 05:55:57AM +0000, WANG Cheng D wrote:
Dear Daniel, As you know, following your suggestion, I can expose shared memory to two containers hosted by the same machine, thus the two containers can exchange data. Now I have a new problem: how does the containers avoid concurrent accesses to the shared memory? In a native system, this can be achieved by using semaphore. But I don't know how the two containers can see the same semaphore. Do I need to design a mechanism to control the access to the shared memory?
Correct, there's no way to share semaphores. Since you hav shared memory though, can you use a POSIX process shared mutex that is allocated in that shared memory block. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (2)
-
Daniel P. Berrange -
WANG Cheng D