Dan Downs (retnuh@retnuh.org) on 8/21/08 wrote:
---------------------------
>I'm trying to look at this from a different direction, or from the side. Basically
>what I'm seeing is that storage size requirements aside the controllers are the
>bottleneck. If I'm connecting 1000 HD or SSD I'd need controllers that can allow
>me to max out the drives potential. So to me the real question is do these controllers exist at any reasonable cost?
>
>If not then I can see why SSDs for this large a scale isn't ready yet, or cost effective.


That is an interesting question. The storage controllers are going to have limits both on maximum throughput and IOPS basis. It's not unreasonable to wonder if an off-the-shelf storage array would max out its (real) IOPS capacity if you replaced all the rotation SAS drives with same sized SSD ones.

And the answer is... I don't know, but it's likely to only miss by a moderate amount.

Most storage controllers can handle far more I/O requests from the hosts than the physical disks in the array can handle, in many cases most of those I/Os are expected to be served out of cache. Second, the net sustainable transfer rates are already much higher than what rotating disk can deliver with random I/Os, and to the extent that random I/Os show little or no throughput penalty on SSDs, that probably won't be an issue.

On the flip side, there is clearly going to be some more overhead in starting many small I/Os than a few bigger (sequential) ones, but don't make the mistake of overestimating the physical size of even sequential I/Os. On the order of 64KB is common. Remember that sequential I/Os will tend to get broken up into chunks of that size for RAID anyway, and really big I/Os that cross stripe boundaries will likely stack multiple I/O requests to each drive. And the main benefit of sequential I/O is in the lack of head movement and rotational latency, so you get those anyway.

Certainly the connection to the physical drive itself is going to be most unaffected, so the issue will be how much extra load the extra random I/Os put on the CPUs in the storage controller.

The connection back to the host might be an issue, as you'll be able to move a lot more data even with random I/Os, and to the extent that the array in question has limits on how much capacity can be installed there, you might have a problem.

But my guess would be that something like a DS3400, which can normally run 48 drives (12 drives in the head DS3400, plus three expansion DS3000s), *might* be limited to 24 or 36 SSD drives (one or two expansion units) before you start hitting performance limits in the head unit. I'd be really surprised if it were worse than that.

On a basic box like a DS3400, you're going to start approaching the (host) I/O bandwidth limits with 48 SSD drives, even with random I/Os - after all you can only put two 4Gb FC interfaces on the thing.

Clearly there will be some amount of tuning and rebalancing of storage controllers for SSDs, but it’s likely to be fairly modest in scope (and more of it will have to do with the SSD write issues than with the reduced latency).