Linus Torvalds (torvalds@linux-foundation.org) on 8/21/08 wrote:
---------------------------
>They're fairly small (64GB is the common size right now),
>and that's probably the biggest issue for most TPC-like
>benchmarks - you just need too many of them for any cheap
>interconnect to really work. You can only do so many SATA
>ports, and hooking them up with some kind of crazy RAID0-
>of-RAID0 SATA concentrators adds cost and enough latency
>to make the thing largely pointless: once you have a
>thousand spindles, you get good IOPS even from much cheaper
>rotational media.
>


While I agree with everything else you said, I don't buy the "complex interconnect hurts you on latency" argument. Adding latency measured in microseconds would hurt you if SSDs had latency figures comparable to DRAM (i.e., measured in nanoseconds) But it sure as hell doesn't matter for storage where latency is typically measured in milliseconds! Good enterprise SSDs ought to be down in the microseconds, but still the interconnect delay won't be a real factor.

If for example you added 50us of latency to 50us of latency that already existed (warning: figures pulled directly from my ass) you would still be talking 10,000 IOPS per device, or about 25x more than the fastest 15k rpm hard drives are capable of. And I'd be hard pressed to figure out how you could design something poorly enough to add 50us when it is effectively just a SATA bridge. Even doing a few levels of store and forward routing on maximum sized packets wouldn't add that much on something clocked at 3 GHz (or 6 for SATA-3)

If you had an SSD capable of 25x more IOPS than a hard drive, you'd need 25x fewer "spindles". Since the storage appears to be overallocated by 2.5x even with 73 GB drives, it appears that 750 GB would be about the 'right' size for an SSD if it could do 10,000 IOPS. It'd be a lot simpler system with 80 drives instead of 2,000, and quite possibly cheaper too even with the premium you pay for flash storage.

I don't know that you can buy a 750 GB SSD anywhere but that's not a technical limitation. I don't know the per device power draw but you could probably do several TB at least before you exceeded the max power draw for SATA.