[libvirt] [RFC] block I/O throttling: how to enable in libvirt
----- Forwarded message from Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> ----- Date: Thu, 1 Sep 2011 11:55:17 +0800 From: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> To: Stefan Hajnoczi <stefanha@gmail.com> Cc: "Daniel P. Berrange" <berrange@redhat.com>, Stefan Hajnoczi <stefanha@gmail.com>, Adam Litke <agl@us.ibm.com>, Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>, QEMU Developers <qemu-devel@nongnu.org>, guijianfeng@cn.fujitsu.com, hutao@cn.fujitsu.com Subject: [RFC] block I/O throttling: how to enable in libvirt Message-ID: <20110901035517.GD16985@f15.cn.ibm.com> References: <CAEH94Li_C=BOe2gV8NyM48njYWMBAo9MTGc1eUOh-Y=cODs6VA@mail.gmail.com> <CAJSP0QW1CPCokX=F5z7y==vn1S4wH0VtOaQ7oj4kC7f7uQM4MQ@mail.gmail.com> <20110830134636.GB29130@aglitke.rchland.ibm.com> <CAJSP0QUHm=y8XJC_KXRg7ufFZt3K_XDDfQb--sxjC+c0GjO8qg@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <CAJSP0QUHm=y8XJC_KXRg7ufFZt3K_XDDfQb--sxjC+c0GjO8qg@mail.gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Xagent-From: wuzhy@linux.vnet.ibm.com X-Xagent-To: wuzhy@linux.vnet.ibm.com X-Xagent-Gateway: vmsdvma.vnet.ibm.com (XAGENTU7 at VMSDVMA) On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote:
Subject: Re: The design choice for how to enable block I/O throttling function in libvirt From: Stefan Hajnoczi <stefanha@gmail.com> To: Adam Litke <agl@us.ibm.com> Cc: libvir-list@redhat.com, "Daniel P. Berrange" <berrange@redhat.com>, Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>, Zhi Yong Wu <zwu.kernel@gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Brightmail-Tracker: AAAAAA== X-Xagent-From: stefanha@gmail.com X-Xagent-To: wuzhy@linux.vnet.ibm.com X-Xagent-Gateway: bldgate.vnet.ibm.com (XAGENTU7 at BLDGATE)
On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke <agl@us.ibm.com> wrote:
On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote:
On Tue, Aug 30, 2011 at 3:55 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
I am trying to enable block I/O throttling function in libvirt. But currently i met some design questions, and don't make sure if we should extend blkiotune to support block I/O throttling or introduce one new libvirt command "blkiothrottle" to cover it or not. If you have some better idea, pls don't hesitate to drop your comments.
A little bit of context: this discussion is about adding libvirt support for QEMU disk I/O throttling.
Thanks for the additional context Stefan.
Today libvirt supports the cgroups blkio-controller, which handles proportional shares and throughput/iops limits on host block devices. blkio-controller does not support network file systems (NFS) or other QEMU remote block drivers (curl, Ceph/rbd, sheepdog) since they are not host block devices. QEMU I/O throttling works with all types of -drive and therefore complements blkio-controller.
The first question that pops into my mind is: Should a user need to understand when to use the cgroups blkio-controller vs. the QEMU I/O throttling method? In my opinion, it would be nice if libvirt had a single interface for block I/O throttling and libvirt would decide which mechanism to use based on the type of device and the specific limits that need to be set.
Yes, I agree it would be simplest to pick the right mechanism, depending on the type of throttling the user wants. More below.
I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation:
http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-t...
From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains?
QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have.
So I think the two are complementary:
If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan,
Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work. Daniel and other maintainers, If you are available, can you make some comments for us?:) Regards, Zhi Yong Wu
Stefan
----- End forwarded message -----
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote:
On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote:
On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke <agl@us.ibm.com> wrote:
On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote:
I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation:
http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-t...
From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains?
QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have.
So I think the two are complementary:
If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan,
Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work.
No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command. Stefan
On Thu, Sep 01, 2011 at 09:11:49AM +0100, Stefan Hajnoczi wrote:
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote:
On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote:
On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke <agl@us.ibm.com> wrote:
On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote:
I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation:
http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-t...
From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains?
QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have.
So I think the two are complementary:
If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan,
Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work.
No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command.
Agreed, the virDomainGetBlkioParameters/virDomainSetBlkioParameters APIs, and blkio virsh command are intended to be a generic interface for setting any block related tuning parameters, regardless of what the underling implementation is. So any use of QEMU I/O throttling features should be added to those APIs/commands. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
From: "Daniel P. Berrange" <berrange@redhat.com> To: Stefan Hajnoczi <stefanha@gmail.com> Cc: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>, libvir-list@redhat.com, agl@us.ibm.com, qemu-devel@nongnu.org, guijianfeng@cn.fujitsu.com, hutao@cn.fujitsu.com, zwu.kernel@gmail.com Subject: Re: [RFC] block I/O throttling: how to enable in libvirt Message-ID: <20110901084934.GA14462@redhat.com> Reply-To: "Daniel P. Berrange" <berrange@redhat.com> References: <20110901050531.GB17963@f15.cn.ibm.com> <20110901081149.GB14245@stefanha-thinkpad.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20110901081149.GB14245@stefanha-thinkpad.localdomain> User-Agent: Mutt/1.5.21 (2010-09-15) X-Scanned-By: MIMEDefang 2.67 on 10.5.11.12 X-Brightmail-Tracker: AAAAAA== X-Xagent-From: berrange@redhat.com X-Xagent-To: wuzhy@linux.vnet.ibm.com X-Xagent-Gateway: emeavsc.vnet.ibm.com (XAGENTU at EMEAVSC)
On Thu, Sep 01, 2011 at 09:11:49AM +0100, Stefan Hajnoczi wrote:
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote:
On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote:
On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke <agl@us.ibm.com> wrote:
On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote:
I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation:
http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-t...
From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains?
QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have.
So I think the two are complementary:
If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan,
Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work.
No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command.
Agreed, the virDomainGetBlkioParameters/virDomainSetBlkioParameters APIs, and blkio virsh command are intended to be a generic interface for setting any block related tuning parameters, regardless of what the underling implementation is. So any use of QEMU I/O throttling features should be added to those APIs/commands.
On Thu, Sep 01, 2011 at 09:49:34AM +0100, Daniel P. Berrange wrote: thanks for your suggestions. Regards, Zhi Yong Wu
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 2011-9-1 16:11, Stefan Hajnoczi wrote:
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote:
On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote:
On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke <agl@us.ibm.com> wrote:
On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote:
I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation:
http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-t...
From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains?
QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have.
So I think the two are complementary:
If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan,
Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work.
No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command.
Agreed. Proportional controlling interfaces and throttling interfaces are all the same cgroup subsystem. So Just extend blkiotune to add new options to support throttling tuning. Thanks, Gui
Stefan
On Fri, Sep 02, 2011 at 09:16:59AM +0800, Gui Jianfeng wrote:
Message-ID: <4E602E8B.6010900@cn.fujitsu.com> Date: Fri, 02 Sep 2011 09:16:59 +0800 From: Gui Jianfeng <guijianfeng@cn.fujitsu.com> User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:6.0) Gecko/20110812 Thunderbird/6.0 MIME-Version: 1.0 To: Stefan Hajnoczi <stefanha@gmail.com>, Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> References: <20110901050531.GB17963@f15.cn.ibm.com> <20110901081149.GB14245@stefanha-thinkpad.localdomain> In-Reply-To: <20110901081149.GB14245@stefanha-thinkpad.localdomain> X-MIMETrack: Itemize by SMTP Server on mailserver/fnst(Release 8.5.1FP4|July 25, 2010) at 2011-09-02 09:15:49, Serialize by Router on mailserver/fnst(Release 8.5.1FP4|July 25, 2010) at 2011-09-02 09:15:52, Serialize complete at 2011-09-02 09:15:52 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=ISO-8859-1 X-detected-operating-system: by eggs.gnu.org: FreeBSD 6.x (1) X-Received-From: 222.73.24.84 Cc: libvir-list@redhat.com, hutao@cn.fujitsu.com, qemu-devel@nongnu.org, zwu.kernel@gmail.com, agl@us.ibm.com Subject: Re: [Qemu-devel] [RFC] block I/O throttling: how to enable in libvirt X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: <qemu-devel.nongnu.org> List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>, <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe> List-Archive: </archive/html/qemu-devel> List-Post: <mailto:qemu-devel@nongnu.org> List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help> List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>, <mailto:qemu-devel-request@nongnu.org?subject=subscribe> X-Mailman-Copy: yes Errors-To: qemu-devel-bounces+wuzhy=linux.vnet.ibm.com@nongnu.org Sender: qemu-devel-bounces+wuzhy=linux.vnet.ibm.com@nongnu.org X-Brightmail-Tracker: AAAAAA== X-Xagent-From: guijianfeng@cn.fujitsu.com X-Xagent-To: wuzhy@linux.vnet.ibm.com X-Xagent-Gateway: vmsdvm9.vnet.ibm.com (XAGENTU at VMSDVM9)
On 2011-9-1 16:11, Stefan Hajnoczi wrote:
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote:
On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote:
On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke <agl@us.ibm.com> wrote:
On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote:
I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation:
http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-t...
From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains?
QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have.
So I think the two are complementary:
If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan,
Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work.
No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command.
Agreed. Proportional controlling interfaces and throttling interfaces are all the same cgroup subsystem. So Just extend blkiotune to add new options to support throttling tuning. Hi, Gui, QEMU block I/O throttling is not relative to cgroup subsystem, i think. anyway, thanks for your sugguests.
Regards, Zhi Yong Wu
Thanks, Gui
Stefan
On 2011-9-2 9:58, Zhi Yong Wu wrote:
On Fri, Sep 02, 2011 at 09:16:59AM +0800, Gui Jianfeng wrote:
Message-ID: <4E602E8B.6010900@cn.fujitsu.com> Date: Fri, 02 Sep 2011 09:16:59 +0800 From: Gui Jianfeng <guijianfeng@cn.fujitsu.com> User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:6.0) Gecko/20110812 Thunderbird/6.0 MIME-Version: 1.0 To: Stefan Hajnoczi <stefanha@gmail.com>, Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> References: <20110901050531.GB17963@f15.cn.ibm.com> <20110901081149.GB14245@stefanha-thinkpad.localdomain> In-Reply-To: <20110901081149.GB14245@stefanha-thinkpad.localdomain> X-MIMETrack: Itemize by SMTP Server on mailserver/fnst(Release 8.5.1FP4|July 25, 2010) at 2011-09-02 09:15:49, Serialize by Router on mailserver/fnst(Release 8.5.1FP4|July 25, 2010) at 2011-09-02 09:15:52, Serialize complete at 2011-09-02 09:15:52 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=ISO-8859-1 X-detected-operating-system: by eggs.gnu.org: FreeBSD 6.x (1) X-Received-From: 222.73.24.84 Cc: libvir-list@redhat.com, hutao@cn.fujitsu.com, qemu-devel@nongnu.org, zwu.kernel@gmail.com, agl@us.ibm.com Subject: Re: [Qemu-devel] [RFC] block I/O throttling: how to enable in libvirt X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: <qemu-devel.nongnu.org> List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>, <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe> List-Archive: </archive/html/qemu-devel> List-Post: <mailto:qemu-devel@nongnu.org> List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help> List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>, <mailto:qemu-devel-request@nongnu.org?subject=subscribe> X-Mailman-Copy: yes Errors-To: qemu-devel-bounces+wuzhy=linux.vnet.ibm.com@nongnu.org Sender: qemu-devel-bounces+wuzhy=linux.vnet.ibm.com@nongnu.org X-Brightmail-Tracker: AAAAAA== X-Xagent-From: guijianfeng@cn.fujitsu.com X-Xagent-To: wuzhy@linux.vnet.ibm.com X-Xagent-Gateway: vmsdvm9.vnet.ibm.com (XAGENTU at VMSDVM9)
On 2011-9-1 16:11, Stefan Hajnoczi wrote:
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote:
On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote:
On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke <agl@us.ibm.com> wrote:
On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote: > I/O throttling can be applied independently to each -drive attached to > a guest and supports throughput/iops limits. For more information on > this QEMU feature and a comparison with blkio-controller, see Ryan > Harper's KVM Forum 2011 presentation:
> http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-t...
From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains?
QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have.
So I think the two are complementary:
If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan,
Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work.
No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command.
Agreed. Proportional controlling interfaces and throttling interfaces are all the same cgroup subsystem. So Just extend blkiotune to add new options to support throttling tuning. Hi, Gui, QEMU block I/O throttling is not relative to cgroup subsystem, i think. anyway, thanks for your sugguests.
Ahh, I misunderstand you before. I thought you mentioned the blkio cgroup throttling interfaces. Ok, I think QEmu I/O throttling has similar semantic with blkio cgroup. So extending blkiotune command is preferred, IMHO. Thanks, Gui
Regards,
Zhi Yong Wu
Thanks, Gui
Stefan
On Thu, Sep 01, 2011 at 09:11:49AM +0100, Stefan Hajnoczi wrote:
Date: Thu, 1 Sep 2011 09:11:49 +0100 From: Stefan Hajnoczi <stefanha@gmail.com> To: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> Message-ID: <20110901081149.GB14245@stefanha-thinkpad.localdomain> References: <20110901050531.GB17963@f15.cn.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20110901050531.GB17963@f15.cn.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 74.125.82.173 Cc: libvir-list@redhat.com, guijianfeng@cn.fujitsu.com, agl@us.ibm.com, qemu-devel@nongnu.org, zwu.kernel@gmail.com, hutao@cn.fujitsu.com Subject: Re: [Qemu-devel] [RFC] block I/O throttling: how to enable in libvirt X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: <qemu-devel.nongnu.org> List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>, <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe> List-Archive: </archive/html/qemu-devel> List-Post: <mailto:qemu-devel@nongnu.org> List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help> List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>, <mailto:qemu-devel-request@nongnu.org?subject=subscribe> X-Mailman-Copy: yes Errors-To: qemu-devel-bounces+wuzhy=linux.vnet.ibm.com@nongnu.org Sender: qemu-devel-bounces+wuzhy=linux.vnet.ibm.com@nongnu.org x-cbid: 11090108-7282-0000-0000-0000010970B0 X-IBM-ISS-SpamDetectors: Score=0; BY=0; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0; ST=0; TS=0; UL=0; ISC= X-IBM-ISS-DetailInfo: BY=3.00000211; HX=3.00000168; KW=3.00000007; PH=3.00000001; SC=3.00000001; SDB=6.00067104; UDB=6.00020326; UTC=2011-09-01 08:12:15 X-Xagent-From: stefanha@gmail.com X-Xagent-To: wuzhy@linux.vnet.ibm.com X-Xagent-Gateway: vmsdvm6.vnet.ibm.com (XAGENTU5 at VMSDVM6)
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote:
On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote:
On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke <agl@us.ibm.com> wrote:
On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote:
I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation:
http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-t...
From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains?
QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have.
So I think the two are complementary:
If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan,
Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work.
No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command. QEMU I/O throttling currently don't support those options of blkiotune, such as --live, --config and --current.If those bps/iops settings are modified, it will immediately take effect.
Regards, Zhi Yong Wu
Stefan
On Fri, Sep 2, 2011 at 3:09 AM, Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> wrote:
On Thu, Sep 01, 2011 at 09:11:49AM +0100, Stefan Hajnoczi wrote:
Date: Thu, 1 Sep 2011 09:11:49 +0100 From: Stefan Hajnoczi <stefanha@gmail.com> To: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> Message-ID: <20110901081149.GB14245@stefanha-thinkpad.localdomain> References: <20110901050531.GB17963@f15.cn.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20110901050531.GB17963@f15.cn.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 74.125.82.173 Cc: libvir-list@redhat.com, guijianfeng@cn.fujitsu.com, agl@us.ibm.com, qemu-devel@nongnu.org, zwu.kernel@gmail.com, hutao@cn.fujitsu.com Subject: Re: [Qemu-devel] [RFC] block I/O throttling: how to enable in libvirt X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: <qemu-devel.nongnu.org> List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>, <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe> List-Archive: </archive/html/qemu-devel> List-Post: <mailto:qemu-devel@nongnu.org> List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help> List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>, <mailto:qemu-devel-request@nongnu.org?subject=subscribe> X-Mailman-Copy: yes Errors-To: qemu-devel-bounces+wuzhy=linux.vnet.ibm.com@nongnu.org Sender: qemu-devel-bounces+wuzhy=linux.vnet.ibm.com@nongnu.org x-cbid: 11090108-7282-0000-0000-0000010970B0 X-IBM-ISS-SpamDetectors: Score=0; BY=0; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0; ST=0; TS=0; UL=0; ISC= X-IBM-ISS-DetailInfo: BY=3.00000211; HX=3.00000168; KW=3.00000007; PH=3.00000001; SC=3.00000001; SDB=6.00067104; UDB=6.00020326; UTC=2011-09-01 08:12:15 X-Xagent-From: stefanha@gmail.com X-Xagent-To: wuzhy@linux.vnet.ibm.com X-Xagent-Gateway: vmsdvm6.vnet.ibm.com (XAGENTU5 at VMSDVM6)
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote:
On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote:
On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke <agl@us.ibm.com> wrote:
On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote:
I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation:
http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-t...
From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains?
QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have.
So I think the two are complementary:
If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan,
Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work.
No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command. QEMU I/O throttling currently don't support those options of blkiotune, such as --live, --config and --current.If those bps/iops settings are modified, it will immediately take effect.
The --live, --config, and --current options are implemented inside libvirt and do not require hypervisor support. Take a look at src/qemu/qemu_driver.c:qemuDomainSetBlkioParameters() to see how these options are implemented for blkiotune today. Stefan
On Fri, Sep 02, 2011 at 09:50:42AM +0100, Stefan Hajnoczi wrote:
Date: Fri, 2 Sep 2011 09:50:42 +0100 Message-ID: <CAJSP0QWc9OcOKxG3jGgYD3r5f=2Fqv3SNVcrARc2Tm0YmP+gyg@mail.gmail.com> Subject: Re: [Qemu-devel] [RFC] block I/O throttling: how to enable in libvirt From: Stefan Hajnoczi <stefanha@gmail.com> To: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> Cc: libvir-list@redhat.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Brightmail-Tracker: AAAAAA== X-Xagent-From: stefanha@gmail.com X-Xagent-To: wuzhy@linux.vnet.ibm.com X-Xagent-Gateway: vmsdvm4.vnet.ibm.com (XAGENTU5 at VMSDVM4)
On Fri, Sep 2, 2011 at 3:09 AM, Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> wrote:
On Thu, Sep 01, 2011 at 09:11:49AM +0100, Stefan Hajnoczi wrote:
Date: Thu, 1 Sep 2011 09:11:49 +0100 From: Stefan Hajnoczi <stefanha@gmail.com> To: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> Message-ID: <20110901081149.GB14245@stefanha-thinkpad.localdomain> References: <20110901050531.GB17963@f15.cn.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20110901050531.GB17963@f15.cn.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 74.125.82.173 Cc: libvir-list@redhat.com, guijianfeng@cn.fujitsu.com, agl@us.ibm.com, qemu-devel@nongnu.org, zwu.kernel@gmail.com, hutao@cn.fujitsu.com Subject: Re: [Qemu-devel] [RFC] block I/O throttling: how to enable in libvirt X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: <qemu-devel.nongnu.org> List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>, <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe> List-Archive: </archive/html/qemu-devel> List-Post: <mailto:qemu-devel@nongnu.org> List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help> List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>, <mailto:qemu-devel-request@nongnu.org?subject=subscribe> X-Mailman-Copy: yes Errors-To: qemu-devel-bounces+wuzhy=linux.vnet.ibm.com@nongnu.org Sender: qemu-devel-bounces+wuzhy=linux.vnet.ibm.com@nongnu.org x-cbid: 11090108-7282-0000-0000-0000010970B0 X-IBM-ISS-SpamDetectors: Score=0; BY=0; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0; ST=0; TS=0; UL=0; ISC= X-IBM-ISS-DetailInfo: BY=3.00000211; HX=3.00000168; KW=3.00000007; PH=3.00000001; SC=3.00000001; SDB=6.00067104; UDB=6.00020326; UTC=2011-09-01 08:12:15 X-Xagent-From: stefanha@gmail.com X-Xagent-To: wuzhy@linux.vnet.ibm.com X-Xagent-Gateway: vmsdvm6.vnet.ibm.com (XAGENTU5 at VMSDVM6)
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote:
On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote:
On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke <agl@us.ibm.com> wrote:
On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote: > I/O throttling can be applied independently to each -drive attached to > a guest and supports throughput/iops limits. For more information on > this QEMU feature and a comparison with blkio-controller, see Ryan > Harper's KVM Forum 2011 presentation:
> http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-t...
From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains?
QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have.
So I think the two are complementary:
If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan,
Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work.
No, I think that the blkiotune command should be extended to support QEMU I/O throttling. This is not new functionality, we already have cgroups blkio-controller support today. Therefore I think it makes sense to keep a unified interface instead of adding a new command. QEMU I/O throttling currently don't support those options of blkiotune, such as --live, --config and --current.If those bps/iops settings are modified, it will immediately take effect.
The --live, --config, and --current options are implemented inside libvirt and do not require hypervisor support. Take a look at src/qemu/qemu_driver.c:qemuDomainSetBlkioParameters() to see how these options are implemented for blkiotune today. I have understood this, thanks.
BTW: i have seen your comments against block I/O throttling patchset, and will respond next week. Regards, Zhi Yong Wu
Stefan
On Thu, Sep 01, 2011 at 01:05:31PM +0800, Zhi Yong Wu wrote:
----- Forwarded message from Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> ----- Date: Thu, 1 Sep 2011 11:55:17 +0800 From: Zhi Yong Wu <wuzhy@linux.vnet.ibm.com> To: Stefan Hajnoczi <stefanha@gmail.com> Cc: "Daniel P. Berrange" <berrange@redhat.com>, Stefan Hajnoczi <stefanha@gmail.com>, Adam Litke <agl@us.ibm.com>, Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>, QEMU Developers <qemu-devel@nongnu.org>, guijianfeng@cn.fujitsu.com, hutao@cn.fujitsu.com Subject: [RFC] block I/O throttling: how to enable in libvirt Message-ID: <20110901035517.GD16985@f15.cn.ibm.com> References: <CAEH94Li_C=BOe2gV8NyM48njYWMBAo9MTGc1eUOh-Y=cODs6VA@mail.gmail.com> <CAJSP0QW1CPCokX=F5z7y==vn1S4wH0VtOaQ7oj4kC7f7uQM4MQ@mail.gmail.com> <20110830134636.GB29130@aglitke.rchland.ibm.com> <CAJSP0QUHm=y8XJC_KXRg7ufFZt3K_XDDfQb--sxjC+c0GjO8qg@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <CAJSP0QUHm=y8XJC_KXRg7ufFZt3K_XDDfQb--sxjC+c0GjO8qg@mail.gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Xagent-From: wuzhy@linux.vnet.ibm.com X-Xagent-To: wuzhy@linux.vnet.ibm.com X-Xagent-Gateway: vmsdvma.vnet.ibm.com (XAGENTU7 at VMSDVMA)
On Wed, Aug 31, 2011 at 08:18:19AM +0100, Stefan Hajnoczi wrote:
Subject: Re: The design choice for how to enable block I/O throttling function in libvirt From: Stefan Hajnoczi <stefanha@gmail.com> To: Adam Litke <agl@us.ibm.com> Cc: libvir-list@redhat.com, "Daniel P. Berrange" <berrange@redhat.com>, Zhi Yong Wu <wuzhy@linux.vnet.ibm.com>, Zhi Yong Wu <zwu.kernel@gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Brightmail-Tracker: AAAAAA== X-Xagent-From: stefanha@gmail.com X-Xagent-To: wuzhy@linux.vnet.ibm.com X-Xagent-Gateway: bldgate.vnet.ibm.com (XAGENTU7 at BLDGATE)
On Tue, Aug 30, 2011 at 2:46 PM, Adam Litke <agl@us.ibm.com> wrote:
On Tue, Aug 30, 2011 at 09:53:33AM +0100, Stefan Hajnoczi wrote:
On Tue, Aug 30, 2011 at 3:55 AM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
I am trying to enable block I/O throttling function in libvirt. But currently i met some design questions, and don't make sure if we should extend blkiotune to support block I/O throttling or introduce one new libvirt command "blkiothrottle" to cover it or not. If you have some better idea, pls don't hesitate to drop your comments.
A little bit of context: this discussion is about adding libvirt support for QEMU disk I/O throttling.
Thanks for the additional context Stefan.
Today libvirt supports the cgroups blkio-controller, which handles proportional shares and throughput/iops limits on host block devices. blkio-controller does not support network file systems (NFS) or other QEMU remote block drivers (curl, Ceph/rbd, sheepdog) since they are not host block devices. QEMU I/O throttling works with all types of -drive and therefore complements blkio-controller.
The first question that pops into my mind is: Should a user need to understand when to use the cgroups blkio-controller vs. the QEMU I/O throttling method? In my opinion, it would be nice if libvirt had a single interface for block I/O throttling and libvirt would decide which mechanism to use based on the type of device and the specific limits that need to be set.
Yes, I agree it would be simplest to pick the right mechanism, depending on the type of throttling the user wants. More below.
I/O throttling can be applied independently to each -drive attached to a guest and supports throughput/iops limits. For more information on this QEMU feature and a comparison with blkio-controller, see Ryan Harper's KVM Forum 2011 presentation:
http://www.linux-kvm.org/wiki/images/7/72/2011-forum-keep-a-limit-on-it-io-t...
From the presentation, it seems that both the cgroups method the the qemu method offer comparable control (assuming a block device) so it might possible to apply either method from the same API in a transparent manner. Am I correct or are we suggesting that the Qemu throttling approach should always be used for Qemu domains?
QEMU I/O throttling does not provide a proportional share mechanism. So you cannot assign weights to VMs and let them receive a fraction of the available disk time. That is only supported by cgroups blkio-controller because it requires a global view which QEMU does not have.
So I think the two are complementary:
If proportional share should be used on a host block device, use cgroups blkio-controller. Otherwise use QEMU I/O throttling. Stefan,
Do you agree with introducing one new libvirt command blkiothrottle now? If so, i will work on the code draft to make it work.
Daniel and other maintainers,
If you are available, can you make some comments for us?:) HI, Adam, Now stefan, Daniel, and Gui all suggest extending blkiotune to keep libivrt unified interface. What do you think of it?
Regards, Zhi Yong Wu
Regards,
Zhi Yong Wu
Stefan
----- End forwarded message -----
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
participants (4)
-
Daniel P. Berrange -
Gui Jianfeng -
Stefan Hajnoczi -
Zhi Yong Wu