Doug Siebert (foo@bar.bar) on 8/21/08 wrote:
>
>While I agree with everything else you said, I don't buy
>the "complex interconnect hurts you on latency" argument.

Hey, I may be overly paranoid, but I've seen too many
"high-end" SCSI controllers that had noticeably sucky
command latency (the same thing goes for SCSI disks, for
that matter).

>If for example you added 50us of latency to 50us of
>latency that already existed (warning: figures pulled
>directly from my ass) you would still be talking 10,000
>IOPS per device

The thing is, latency and IOPS aren't related very much.

A lot of the controllers are pipelining the commands, so
they can have tons of commands in flight, but that does
not mean that the latency for any particular command is
good, even if they get 10,000 IOPS.

You could have a 1 second latency per command, if
your command queue was just deep enough. Performance would
absolutely suck, but you could still claim lots of IOPS
per second, because you'd still complete that many in the
steady state. They'd just all have to be a second old ;)

>Even doing a few levels of store and forward routing on
>maximum sized packets wouldn't add that much on something
>clocked at 3 GHz (or 6 for SATA-3)

I am actually much less concerned about the data
latency than about the command latency.

The data transfers are always done as simple buffers and
DMA from the buffer to the channel, and yes, that transfer
is obviously limited by the channel itself, and almost
always that is not the limiting factor at all (even
if next-gen SSD's are apparently going to start coming
very close to being able to saturate SATA-2).

No, the slow thing is often the command sequencer. There's
one on the controller, there's one on the disk, and there
tends to be at least one translating one for all those
more complex buses.

Do you know what happened with early SATA devices? There
weren't enough SATA disks made, so vendors would basically
reuse all the electronics for a PATA device, and then have
a small micro-controller that translates from one to the
other. And trust me, that micro-controller isn't running at
3GHz.

Do you really think that people sell enough FC disks
that they do FC "natively"? I personally don't.

Same goes for any RAID enclosure or controller. It
serializes all the commands, buffers them up per channel,
and does a fair amount of computation too if we're talking
higher raid levels (ie then it needs to buffer the data
too). It may well be able to saturate the channel, and
not lose any IOPS (in fact, hopefully do more aggregate
IOPS than the single channels it is doing RAID on), but
it's going to add latency.

But you may be right that it's not quite noticeable enough
to matter.

But I'd like to point out the stories from SSD manufacturers
about how OS's aren't getting everything they should from
SSD performance. How much do you want to bet it's because
of the fact that the IO latency dropped by two orders of
magnitude, and all the optimization decisions turn out to
be wrong?

And I'm just pretty sure the same thing is true when it
comes to things like existing FC controllers.

Linus