[Libvir] [PATCH] Xen: Support cpu_weight and cpu_cap for Xen.
Hi, I set the information of cpu_weight/cpu_cap to configuration file or sxp format, and execute "virsh start". But the information of weight/cap is lost. libvirt doesn't support cpu_weight and cpu_cap for XML format. see also Bz#337591 https://bugzilla.redhat.com/show_bug.cgi?id=337591 I make a patch to add weight/cap attributes as vcpus element (eg. <vcpus weight='512' cap='280'>4</vcpus>) c.f. the thread at: https://www.redhat.com/archives/libvir-list/2007-October/msg00044.html Signed-off-by: Tatsuro Enokura <fj7716hz@aa.jp.fujitsu.com> Thanks, Tatsuro Enokura Index: libvirt/src/xend_internal.c =================================================================== RCS file: /data/cvs/libvirt/src/xend_internal.c,v retrieving revision 1.151 diff -u -p -r1.151 xend_internal.c --- libvirt/src/xend_internal.c 22 Oct 2007 20:28:55 -0000 1.151 +++ libvirt/src/xend_internal.c 24 Oct 2007 06:02:27 -0000 @@ -1438,8 +1438,19 @@ xend_parse_sexp_desc(virConnectPtr conn, if ((cur_mem >= MIN_XEN_GUEST_SIZE) && (cur_mem != max_mem)) virBufferVSprintf(&buf, " <currentMemory>%d</currentMemory>\n", cur_mem); - virBufferVSprintf(&buf, " <vcpu>%d</vcpu>\n", + virBufferVSprintf(&buf, " <vcpu"); + if (sexpr_node(root, "domain/cpu_weight") != NULL) { + virBufferVSprintf(&buf, " weight='%d'", + sexpr_int(root, "domain/cpu_weight")); + } + if (sexpr_node(root, "domain/cpu_cap") != NULL) { + virBufferVSprintf(&buf, " cap='%d'", + sexpr_int(root, "domain/cpu_cap")); + } + virBufferVSprintf(&buf, ">%d</vcpu>\n", sexpr_int(root, "domain/vcpus")); + + tmp = sexpr_node(root, "domain/on_poweroff"); if (tmp != NULL) virBufferVSprintf(&buf, " <on_poweroff>%s</on_poweroff>\n", tmp); Index: libvirt/src/xm_internal.c =================================================================== RCS file: /data/cvs/libvirt/src/xm_internal.c,v retrieving revision 1.41 diff -u -p -r1.41 xm_internal.c --- libvirt/src/xm_internal.c 10 Oct 2007 17:55:38 -0000 1.41 +++ libvirt/src/xm_internal.c 24 Oct 2007 06:02:28 -0000 @@ -661,9 +661,14 @@ char *xenXMDomainFormatXML(virConnectPtr virBufferVSprintf(buf, " <memory>%ld</memory>\n", val * 1024); + virBufferVSprintf(buf, " <vcpu"); + if (xenXMConfigGetInt(conf, "cpu_weight", &val) == 0) + virBufferVSprintf(buf, " weight='%ld'", val); + if (xenXMConfigGetInt(conf, "cpu_cap", &val) == 0) + virBufferVSprintf(buf, " cap='%ld'", val); if (xenXMConfigGetInt(conf, "vcpus", &val) < 0) val = 1; - virBufferVSprintf(buf, " <vcpu>%ld</vcpu>\n", val); + virBufferVSprintf(buf, ">%ld</vcpu>\n", val); if (xenXMConfigGetString(conf, "on_poweroff", &str) < 0) @@ -1820,6 +1825,12 @@ virConfPtr xenXMParseXMLToConfig(virConn if (xenXMConfigSetIntFromXPath(conn, conf, ctxt, "vcpus", "string(/domain/vcpu)", 0, 1, "cannot set vcpus config parameter") < 0) goto error; + if (xenXMConfigSetIntFromXPath(conn, conf, ctxt, "cpu_weight", "string(/domain/vcpu/@weight)", 0, 1, + "cannot set cpu_weight config paramerter") < 0) + goto error; + if (xenXMConfigSetIntFromXPath(conn, conf, ctxt, "cpu_cap", "string(/domain/vcpu/@cap)", 0, 1, + "cannot set cpu_cap config paramerter") < 0) + goto error; obj = xmlXPathEval(BAD_CAST "string(/domain/os/type)", ctxt); if ((obj != NULL) && (obj->type == XPATH_STRING) && Index: libvirt/src/xml.c =================================================================== RCS file: /data/cvs/libvirt/src/xml.c,v retrieving revision 1.94 diff -u -p -r1.94 xml.c --- libvirt/src/xml.c 23 Oct 2007 15:31:33 -0000 1.94 +++ libvirt/src/xml.c 24 Oct 2007 06:02:28 -0000 @@ -1529,6 +1529,12 @@ virDomainParseXMLDesc(virConnectPtr conn vcpus = (unsigned int) f; } virBufferVSprintf(&buf, "(vcpus %u)", vcpus); + if (virXPathNumber("number(/domain/vcpu[1]/@weight)", ctxt, &f) == 0) { + virBufferVSprintf(&buf, "(cpu_weight %ld)", (long) f); + } + if (virXPathNumber("number(/domain/vcpu[1]/@cap)", ctxt, &f) == 0) { + virBufferVSprintf(&buf, "(cpu_cap %ld)", (long) f); + } str = virXPathString("string(/domain/vcpu/@cpuset)", ctxt); if (str != NULL) {
On Wed, Oct 24, 2007 at 04:35:33PM +0900, Tatsuro Enokura wrote:
Hi,
I set the information of cpu_weight/cpu_cap to configuration file or sxp format, and execute "virsh start". But the information of weight/cap is lost. libvirt doesn't support cpu_weight and cpu_cap for XML format.
Right. To me cpu_weight and cpu_cap are tuning informations they are completely dependant: - on the hypervisor used (only Xen) - on a specific scheduler for said hypervisor - on the runtime operations conditions, i.e. you may want to revise those settings if you migrate the domain, or change anything on the target host So you are suggesting to add this to the XML, while for me this makes little sense because of all this specificity. I have no doubt the patch would 'work' for you, but anybody using a different hypervisor, or different scheduler, or even someone trying to understand what those fields are would have no use or informations (your patch does not provide documentation for the meaning of those attributes). I have a problem with extending the XML in a way which makes sense only for one hypervisor, when using a specific scheduler, and without a proper definition for what the extension actually means. Also Those informations are highly runtime dependant, it's tuning, it is not critical at all to get that tuning to get the domain up and running, and once it is running you can actually use the libvirt API to make the scheduler tuning. Can you explain why you absolutely want to have that tuning information in the XML itself ?
see also Bz#337591 https://bugzilla.redhat.com/show_bug.cgi?id=337591
That's your request, not really a justification
I make a patch to add weight/cap attributes as vcpus element (eg. <vcpus weight='512' cap='280'>4</vcpus>) c.f. the thread at: https://www.redhat.com/archives/libvir-list/2007-October/msg00044.html
That thread was absolutely not discussing specific Xen scheduling parameters it was discussing how to express the set of CPUs a domain should start on and this information is not Xen specific in nature. Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
Hi, Daniel Daniel Veillard wrote:
I set the information of cpu_weight/cpu_cap to configuration file or sxp format, and execute "virsh start". But the information of weight/cap is lost. libvirt doesn't support cpu_weight and cpu_cap for XML format.
Right. To me cpu_weight and cpu_cap are tuning informations they are completely dependant: - on the hypervisor used (only Xen) - on a specific scheduler for said hypervisor - on the runtime operations conditions, i.e. you may want to revise those settings if you migrate the domain, or change anything on the target host
So you are suggesting to add this to the XML, while for me this makes little sense because of all this specificity. I have no doubt the patch would 'work' for you, but anybody using a different hypervisor, or different scheduler, or even someone trying to understand what those fields are would have no use or informations (your patch does not provide documentation for the meaning of those attributes).
I have a problem with extending the XML in a way which makes sense only for one hypervisor, when using a specific scheduler, and without a proper definition for what the extension actually means.
Yes, scheduler funtcion is xen specific. Other hypervisor(like qemu, kvm...) hasn't scheduer function. Threfore, extending weight/cap for the XML is meaningless for other hypervisor. The libvirt scheduler API is xen specific, it is necessary to add for XML's documentation that attributes weight/cap is effective only in xen.
Also Those informations are highly runtime dependant, it's tuning, it is not critical at all to get that tuning to get the domain up and running, and once it is running you can actually use the libvirt API to make the scheduler tuning.
Can you explain why you absolutely want to have that tuning information in the XML itself ?
In xentool case, weight/cap set by user to configuration file/sxp is seting at starting the domain. The user may not do anything. In libvirt case, the user should set weight/cap using libvirt api after starting the domain. I think that it is not convenient for the user. Thanks, Tatsuro Enokura
Daniel Veillard wrote:
So you are suggesting to add this to the XML, while for me this makes little sense because of all this specificity. I have no doubt the patch would 'work' for you, but anybody using a different hypervisor, or different scheduler, or even someone trying to understand what those fields are would have no use or informations (your patch does not provide documentation for the meaning of those attributes).
I have a problem with extending the XML in a way which makes sense only for one hypervisor, when using a specific scheduler, and without a proper definition for what the extension actually means. Also Those informations are highly runtime dependant, it's tuning, it is not critical at all to get that tuning to get the domain up and running, and once it is running you can actually use the libvirt API to make the scheduler tuning.
Can you explain why you absolutely want to have that tuning information in the XML itself ?
There's certainly a general problem here: how do we persist information like CPU pinning and scheduler information across domain shutdown to the next time that the domain starts up. What is the current thinking about setting CPU pinning info when a domain boots? Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903
On Fri, Oct 26, 2007 at 12:15:38PM +0100, Richard W.M. Jones wrote:
Daniel Veillard wrote:
So you are suggesting to add this to the XML, while for me this makes little sense because of all this specificity. I have no doubt the patch would 'work' for you, but anybody using a different hypervisor, or different scheduler, or even someone trying to understand what those fields are would have no use or informations (your patch does not provide documentation for the meaning of those attributes).
I have a problem with extending the XML in a way which makes sense only for one hypervisor, when using a specific scheduler, and without a proper definition for what the extension actually means. Also Those informations are highly runtime dependant, it's tuning, it is not critical at all to get that tuning to get the domain up and running, and once it is running you can actually use the libvirt API to make the scheduler tuning.
Can you explain why you absolutely want to have that tuning information in the XML itself ?
There's certainly a general problem here: how do we persist information like CPU pinning and scheduler information across domain shutdown to the next time that the domain starts up. What is the current thinking about setting CPU pinning info when a domain boots?
To me that's the same thing, CPU pinning and scheduler information are basically tuning informations. Maybe this is static, maybe this needs to change frequently, but we cannot assume the first case for the design. What would make more sense to me is to be able to save and restore tuning informations, per domain or for the full system at the virsh level. For example something like: virsh savetuning > tuning.data or virsh savetuning domain > tuning.data and virsh loadtuning tuning.data This would still be convenient for the user, especially for locked systems or systems where multiple different workload can be used depending on the demand, one would just need to save tuning for each of them. Tuning could be reloaded or readjusted ech time a domain is stopped/started or the whole node is rebooted. The implementation could be done entierely in libvirt without having to change or extend existing APIs or behaviour. I can understand the need to make it easy for an user, I still don't think this means those tuning informations need to be associated to the domain definition, to me it is somehow orthogonal to the domain themselves and I would rather try to provide a good solution to the problem, than try to imitate how Xen was doing that. Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Fri, 2007-10-26 at 09:08 -0400, Daniel Veillard wrote:
I can understand the need to make it easy for an user, I still don't think this means those tuning informations need to be associated to the domain definition, to me it is somehow orthogonal to the domain themselves and I would rather try to provide a good solution to the problem, than try to imitate how Xen was doing that.
Surely, users who set tuning parameters view them as part of their domain description. To me, having to keep track of yet another file to have the complete description of a domain isn't very appealing. There's plenty of people that only use one hypervisor, and for them, that something only works on Xen or kvm is irrelevant. David
David Lutterkort wrote:
On Fri, 2007-10-26 at 09:08 -0400, Daniel Veillard wrote:
I can understand the need to make it easy for an user, I still don't think this means those tuning informations need to be associated to the domain definition, to me it is somehow orthogonal to the domain themselves and I would rather try to provide a good solution to the problem, than try to imitate how Xen was doing that.
Surely, users who set tuning parameters view them as part of their domain description.
I agree. It appears the discussions around recent Linux-VServer patches includes tuning information in the XML description so I'm a little confused if this idea is being received positively or not. I would like to avoid calling out to some other API for scheduling parameters at domU definition if at all possible. Perhaps I've been away too long and this functionality already exists :-), but not seeing it in http://www.libvirt.org/format.html Regards, Jim
I am not a user, so my feelings here may be incorrect, but it seems it might be unpleasant to have to modify 30 (for example) domain configs when I'm trying to rework tuning information for a machine. I could imagine a single config file that contains cpu_weight, cpu_cap, the cpuset, and whatever other tuning information per domain might be appropriate. Looking at a single file to tweak such values across multiple domains might be preferable, and make the situation easier to visualize. Jim Fehlig wrote:
David Lutterkort wrote:
On Fri, 2007-10-26 at 09:08 -0400, Daniel Veillard wrote:
I can understand the need to make it easy for an user, I still don't think this means those tuning informations need to be associated to the domain definition, to me it is somehow orthogonal to the domain themselves and I would rather try to provide a good solution to the problem, than try to imitate how Xen was doing that.
Surely, users who set tuning parameters view them as part of their domain description.
I agree. It appears the discussions around recent Linux-VServer patches includes tuning information in the XML description so I'm a little confused if this idea is being received positively or not.
I would like to avoid calling out to some other API for scheduling parameters at domU definition if at all possible. Perhaps I've been away too long and this functionality already exists :-), but not seeing it in http://www.libvirt.org/format.html
Regards, Jim
-- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-- Elizabeth Kon (Beth) IBM Linux Technology Center Open Hypervisor Team email: eak@us.ibm.com
Daniel Veillard wrote:
On Fri, Oct 26, 2007 at 12:15:38PM +0100, Richard W.M. Jones wrote:
So you are suggesting to add this to the XML, while for me this makes little sense because of all this specificity. I have no doubt the patch would 'work' for you, but anybody using a different hypervisor, or different scheduler, or even someone trying to understand what those fields are would have no use or informations (your patch does not provide documentation for the meaning of those attributes).
I have a problem with extending the XML in a way which makes sense only for one hypervisor, when using a specific scheduler, and without a proper definition for what the extension actually means. Also Those informations are highly runtime dependant, it's tuning, it is not critical at all to get that tuning to get the domain up and running, and once it is running you can actually use the libvirt API to make the scheduler tuning.
Can you explain why you absolutely want to have that tuning information in the XML itself ? There's certainly a general problem here: how do we persist information
Daniel Veillard wrote: like CPU pinning and scheduler information across domain shutdown to the next time that the domain starts up. What is the current thinking about setting CPU pinning info when a domain boots?
To me that's the same thing, CPU pinning and scheduler information are basically tuning informations. Maybe this is static, maybe this needs to change frequently, but we cannot assume the first case for the design.
What would make more sense to me is to be able to save and restore tuning informations, per domain or for the full system at the virsh level. For example something like: virsh savetuning > tuning.data or virsh savetuning domain > tuning.data
and
virsh loadtuning tuning.data
This would still be convenient for the user, especially for locked systems or systems where multiple different workload can be used depending on the demand, one would just need to save tuning for each of them. Tuning could be reloaded or readjusted ech time a domain is stopped/started or the whole node is rebooted. The implementation could be done entierely in libvirt without having to change or extend existing APIs or behaviour.
I can understand the need to make it easy for an user, I still don't think this means those tuning informations need to be associated to the domain definition, to me it is somehow orthogonal to the domain themselves and I would rather try to provide a good solution to the problem, than try to imitate how Xen was doing that.
[Resurrecting this old thread ...] How about this as a plan? (1) Add 'virsh savetuning' and 'virsh loadtuning' commands as Daniel Veillard has suggested above. These would save the per-domain tuning information to an XML file <tuning> <pin vcpu="0" pcpu="0"/> <pin vcpu="1" pcpu="1"/> <!-- other stuff for cpu_weight, etc. --> </tuning> (2) Modify 'virsh define' and 'virsh create' commands to add a '--with-tuning' flag, so that: virsh define foo.xml --with-tuning foo-tuning.xml Which basically does the ordinary 'virsh define' step followed by 'virsh loadtuning'. (3) Modify 'virsh save' to add '--save-tuning', so that: virsh save foo foo.img --save-tuning foo-tuning.xml This captures the tuning information into the XML file, and is the same as doing an ordinary 'save' followed by 'virsh savetuning'. - - - This doesn't change the libvirt API, nor does it deal with saving and restoring tuning information across restarts (but then I can't see how the autostart flag works for Xen at the moment at all ... a bug?) I'm prepared to do this work if people think it's a good idea. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903
On Tue, Dec 18, 2007 at 02:53:09PM +0000, Richard W.M. Jones wrote:
Daniel Veillard wrote:
On Fri, Oct 26, 2007 at 12:15:38PM +0100, Richard W.M. Jones wrote:
So you are suggesting to add this to the XML, while for me this makes little sense because of all this specificity. I have no doubt the patch would 'work' for you, but anybody using a different hypervisor, or different scheduler, or even someone trying to understand what those fields are would have no use or informations (your patch does not provide documentation for the meaning of those attributes).
I have a problem with extending the XML in a way which makes sense only for one hypervisor, when using a specific scheduler, and without a proper definition for what the extension actually means. Also Those informations are highly runtime dependant, it's tuning, it is not critical at all to get that tuning to get the domain up and running, and once it is running you can actually use the libvirt API to make the scheduler tuning.
Can you explain why you absolutely want to have that tuning information in the XML itself ? There's certainly a general problem here: how do we persist information
Daniel Veillard wrote: like CPU pinning and scheduler information across domain shutdown to the next time that the domain starts up. What is the current thinking about setting CPU pinning info when a domain boots?
To me that's the same thing, CPU pinning and scheduler information are basically tuning informations. Maybe this is static, maybe this needs to change frequently, but we cannot assume the first case for the design.
What would make more sense to me is to be able to save and restore tuning informations, per domain or for the full system at the virsh level. For example something like: virsh savetuning > tuning.data or virsh savetuning domain > tuning.data
and
virsh loadtuning tuning.data
This would still be convenient for the user, especially for locked systems or systems where multiple different workload can be used depending on the demand, one would just need to save tuning for each of them. Tuning could be reloaded or readjusted ech time a domain is stopped/started or the whole node is rebooted. The implementation could be done entierely in libvirt without having to change or extend existing APIs or behaviour.
I can understand the need to make it easy for an user, I still don't think this means those tuning informations need to be associated to the domain definition, to me it is somehow orthogonal to the domain themselves and I would rather try to provide a good solution to the problem, than try to imitate how Xen was doing that.
[Resurrecting this old thread ...]
How about this as a plan?
(1) Add 'virsh savetuning' and 'virsh loadtuning' commands as Daniel Veillard has suggested above. These would save the per-domain tuning information to an XML file
<tuning> <pin vcpu="0" pcpu="0"/> <pin vcpu="1" pcpu="1"/> <!-- other stuff for cpu_weight, etc. --> </tuning>
I have early code but never commited it, it doesn't do much yet, but yes that was basicaly my plan
(2) Modify 'virsh define' and 'virsh create' commands to add a '--with-tuning' flag, so that:
virsh define foo.xml --with-tuning foo-tuning.xml
Which basically does the ordinary 'virsh define' step followed by 'virsh loadtuning'.
(3) Modify 'virsh save' to add '--save-tuning', so that:
virsh save foo foo.img --save-tuning foo-tuning.xml
This captures the tuning information into the XML file, and is the same as doing an ordinary 'save' followed by 'virsh savetuning'.
- - -
yeah, I think it's reasonnable since it doesn't change the C api and is clearly a virsh extension.
This doesn't change the libvirt API, nor does it deal with saving and restoring tuning information across restarts (but then I can't see how the autostart flag works for Xen at the moment at all ... a bug?)
I'm prepared to do this work if people think it's a good idea.
Sounds fine to me, ideally make the virsh extension as a separate module to allow easy copying in other user's applications. I had defined: int virDomainSaveTuningTo(xmlBufferPtr buf, virDomainPtr dom, int flags) and char * virConnSaveTuning(virConnectPtr conn, int flags) which would be fine as separate entry points if someone wanted to embbed them. Of course this fode is 10x easier to write in Python or something else dealing easilly with strings ... Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
Hi, Richard Richard W.M. Jones wrote:
[Resurrecting this old thread ...]
How about this as a plan?
(1) Add 'virsh savetuning' and 'virsh loadtuning' commands as Daniel Veillard has suggested above. These would save the per-domain tuning information to an XML file
<tuning> <pin vcpu="0" pcpu="0"/> <pin vcpu="1" pcpu="1"/> <!-- other stuff for cpu_weight, etc. --> </tuning>
Ok, looks fine to me.
(2) Modify 'virsh define' and 'virsh create' commands to add a '--with-tuning' flag, so that:
virsh define foo.xml --with-tuning foo-tuning.xml
Which basically does the ordinary 'virsh define' step followed by 'virsh loadtuning'.
I agree and I want to add '--with-tuning' flag to 'virsh start' command, too.
(3) Modify 'virsh save' to add '--save-tuning', so that:
virsh save foo foo.img --save-tuning foo-tuning.xml
This captures the tuning information into the XML file, and is the same as doing an ordinary 'save' followed by 'virsh savetuning'.
Sounds good to me. Thanks, Tatsuro Enokura
Tatsuro Enokura wrote:
Hi, Richard
Richard W.M. Jones wrote:
[Resurrecting this old thread ...]
How about this as a plan?
(1) Add 'virsh savetuning' and 'virsh loadtuning' commands as Daniel Veillard has suggested above. These would save the per-domain tuning information to an XML file
<tuning> <pin vcpu="0" pcpu="0"/> <pin vcpu="1" pcpu="1"/> <!-- other stuff for cpu_weight, etc. --> </tuning>
Ok, looks fine to me.
(2) Modify 'virsh define' and 'virsh create' commands to add a '--with-tuning' flag, so that:
virsh define foo.xml --with-tuning foo-tuning.xml
Which basically does the ordinary 'virsh define' step followed by 'virsh loadtuning'.
I agree and I want to add '--with-tuning' flag to 'virsh start' command, too.
(3) Modify 'virsh save' to add '--save-tuning', so that:
virsh save foo foo.img --save-tuning foo-tuning.xml
This captures the tuning information into the XML file, and is the same as doing an ordinary 'save' followed by 'virsh savetuning'.
Sounds good to me.
OK good. I probably won't get around to this real soon now because I've got a huge list of things on my to do list, but if the design is good I can come up with an implementation in a week or two. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903
participants (6)
-
beth kon -
Daniel Veillard -
David Lutterkort -
Jim Fehlig -
Richard W.M. Jones -
Tatsuro Enokura