large number of hot plug disks
Hi, I am working on a project, and one of the requirements is that I can hotplug a large number of disks. Think several dozens to potentially 100+. The guest OS will be some linux flavor so it should have no problem with a large number of disks. During my experimentation I quickly ran into the "No available PCI slots" limit. I found https://libvirt.org/pci-hotplug.html explaining to me why the limit is there. Now I could add a bunch of pcie-root-ports ahead of time, but there is likely a limit to that as well. So before I go down some crazy paths, I figured I would ask on here if there is a better plan than adding a bunch of root ports for adding a large number of hotplugged disks? Thanks, Alexander
On Tue, Sep 08, 2020 at 08:19:21 -0400, Alexander Wels wrote:
Hi,
I am working on a project, and one of the requirements is that I can hotplug a large number of disks. Think several dozens to potentially 100+. The guest OS will be some linux flavor so it should have no problem with a large number of disks.
During my experimentation I quickly ran into the "No available PCI slots" limit. I found https://libvirt.org/pci-hotplug.html explaining to me why the limit is there. Now I could add a bunch of pcie-root-ports ahead of time, but there is likely a limit to that as well. So before I go down some crazy paths, I figured I would ask on here if there is a better plan than adding a bunch of root ports for adding a large number of hotplugged disks?
I strongly suggest using a virtio-scsi controller and plugging disks into it. You can have a fairly large number of LUNs per controller, so that will severely decrease the amount of required PCI slots. You can obviously add more controllers if you need even more LUNs.
On Tue, Sep 08, 2020 at 02:38:17PM +0200, Peter Krempa wrote:
On Tue, Sep 08, 2020 at 08:19:21 -0400, Alexander Wels wrote:
Hi,
I am working on a project, and one of the requirements is that I can hotplug a large number of disks. Think several dozens to potentially 100+. The guest OS will be some linux flavor so it should have no problem with a large number of disks.
During my experimentation I quickly ran into the "No available PCI slots" limit. I found https://libvirt.org/pci-hotplug.html explaining to me why the limit is there. Now I could add a bunch of pcie-root-ports ahead of time, but there is likely a limit to that as well. So before I go down some crazy paths, I figured I would ask on here if there is a better plan than adding a bunch of root ports for adding a large number of hotplugged disks?
I strongly suggest using a virtio-scsi controller and plugging disks into it. You can have a fairly large number of LUNs per controller, so that will severely decrease the amount of required PCI slots. You can obviously add more controllers if you need even more LUNs.
A single virtio-scsi controller can support ~4 million LUNs. IOW you'll face a wall of other problems before you get anywhere near needing a second virtio-scsi controller due to LUN exhaustion :-) Only real reason for multiple virtio-scsi controllers that I see is if you want to setup different host CPU affinity for each controller for I/O thread performance tuning Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Sep 08, 2020 at 08:19:21AM -0400, Alexander Wels wrote:
Hi,
I am working on a project, and one of the requirements is that I can hotplug a large number of disks. Think several dozens to potentially 100+. The guest OS will be some linux flavor so it should have no problem with a large number of disks.
During my experimentation I quickly ran into the "No available PCI slots" limit. I found https://libvirt.org/pci-hotplug.html explaining to me why the limit is there. Now I could add a bunch of pcie-root-ports ahead of time, but there is likely a limit to that as well. So before I go down some crazy paths, I figured I would ask on here if there is a better plan than adding a bunch of root ports for adding a large number of hotplugged disks?
While you can add extra PCI bridges, I'd suggest using virtio-scsi instead These blog posts should serve as warning about some of the limits you'll hit: https://rwmj.wordpress.com/2017/04/25/how-many-disks-can-you-add-to-a-virtua... https://rwmj.wordpress.com/2017/04/28/how-many-disks-can-you-add-to-a-virtua... Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Sep 8, 2020 at 8:43 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Sep 08, 2020 at 08:19:21AM -0400, Alexander Wels wrote:
Hi,
I am working on a project, and one of the requirements is that I can hotplug a large number of disks. Think several dozens to potentially 100+. The guest OS will be some linux flavor so it should have no problem with a large number of disks.
During my experimentation I quickly ran into the "No available PCI slots" limit. I found https://libvirt.org/pci-hotplug.html explaining to me why the limit is there. Now I could add a bunch of pcie-root-ports ahead of time, but there is likely a limit to that as well. So before I go down some crazy paths, I figured I would ask on here if there is a better plan than adding a bunch of root ports for adding a large number of hotplugged disks?
While you can add extra PCI bridges, I'd suggest using virtio-scsi instead
These blog posts should serve as warning about some of the limits you'll hit:
https://rwmj.wordpress.com/2017/04/25/how-many-disks-can-you-add-to-a-virtua...
https://rwmj.wordpress.com/2017/04/28/how-many-disks-can-you-add-to-a-virtua...
Thanks I read those. my absolute upper limit is going to be in the 100ish range so I should be good there. I will investigate virtio-scsi.
Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (3)
-
Alexander Wels -
Daniel P. Berrangé -
Peter Krempa