Doug Siebert (foo@bar.bar) on 9/11/08 wrote:
>
>Regular RAM and a small (pre-erased) reserved section of
>flash along with a capacitor would be a much better
>solution.

If you have pre-erased flash, why bother with the whole
charade to begin with?

Guys, this is basic queuing theory. You cannot get write
bandwidth higher than what your flash is able to absorb
(and that includes all the GC and erase cycles necessary)
in the steady state anyway.

No amount of buffering will ever change that.

Anybody who thinks that they can lower latency by just
adding buffers is a moron, because the fact is,
if you "lie" and tell the OS that the write has completed,
it will just write more. Until your buffer is full, and
you have to throttle the writes!

And at that point, you have to expose the real latency
(or worse - much bigger latencies because you then
end up waiting for all of the buffering you did!)

Buffering can only help temporary spikes.

And there is almost certainly no reason for those
temporary spikes, except for simplistic initial SSD
controller implementations.

Once you are clever and careful enough to try to do any
capacitor that writes back after power failure, you're
already spending more effort and engineering (on a hacky
workaround) than if you were just doing incremental GC and
compaction in the first place.

Because once you do incremental compaction and GC, you'll
not need that special reserved pre-erased part of
the flash - because you just make sure that your compaction
and GC stays just ahead of the curve.

That's what realtime GC means for chissake!

So you're all trying to solve the wrong problem. You're
looking at badly designed SSD's (in fairness, it always
takes a few generations for people to learn even the basic
"tricks", so "badly designed" is a bit harsh), and then
you are trying to make these insane workarounds for it,
when the right way to solve it is to just do it better
in the first place, and suddenly the workarounds aren't
needed any more.

See what I'm saying?

Let me put this another way, if it's unclear:

- you can never fundamentally lower the average latency
for one operation to below that of the average
throughput for that size operation of the device.

Think about it.

If you could make the write latency lower than the
throughput, the thing that feeds it would just feed it
more, and if you can absorb that, then your throughput
would be higher than you just started out claiming it
was.

- Ergo: assuming latencies are fairly stable (ie the
latencies aren't sometimes low and sometimes very high),
that means that you can never realistically win
by buffering more than the data you can write in one
such "latency time".

- Thinking that you can just make buffers bigger is silly.
The fundamental buffer size is going to be "throughput
times latency time". No more, no less. (Of course, for
implementation reasons you might want to do things like
switch buffers around so you'd do double buffering, but
we're talking about just small multipliers here).

- And people know how to do realtime GC and
compaction.

Another way of saying the same thing: in the absense of
things like mechanical arm movement etc that introduces
big latency jumps, it sure as h*ll should be entirely
possible to keep latencies stable. You don't need
to have events that suddenly rewrite big portions of the
disk in order to do GC and erase-cycles!

- once your buffers are in the ballpark of the above
"basic size", then trying to return success early is
not going to help anything anyway. If you lie and return
early (and have a complex battery back-up to protect you
from the downsides of lying), you're just going to be
stuck ont he next IO instead.

So having some kind of super-capacitor is fundamentally
stupid. The "claim the IO is done early" is just stupid.
Much better to not lie, and thus not need battery
backup.

So why do people do high-performance controllers that
really do do battery backup etc? I just told you
it was stupid.

A couple of reasons:

- you can absorb load spikes. The big buffers
won't help you under heavy load, but it will help
you under spiky load.

- for rotational media, you can absorb the much longer
latencies for actual seek events. This is especially
true for RAID controllers that have huge bandwidth
over multiple disks (so the "latency times throughput"
number is actually reasonably large).

- again, mainly for rotational media: you can sort the
final requests and actually improve throughput by doing
the IO in a different order. IOW, you can actually
decrease the latencies by seeking less, but you need
lots of data to do that well.

But apart from the first one, those reasons are largely not
valid for solid-state media. The first one is true, but
quite frankly, it's much better done somewhere else
than on the disk itself. IOW, you're better off doing it
on the host controller, where the fundamental latencies are
lower, so you get a bigger win (ie for bursty IO, you may
be getting a fake latency to the host controller, not the
one all the way to the disk).

Doing it on the host controller is better for another
reason too: it allows you basic building block (the disk)
to be generic and cheap. Which is what you want. Because
people are going to buy cheap, and the people who want
something extra and are willing to pay for it, are still
better off if they can use mass-produced basic building
blocks and then just adding some "secret sauce".

(That "standard building block" detail is true for the
other cases too, for that matter, and probably explains
why nobody I know of actually does a disk with battery
back-up).

To recap:

- doing large buffers is stupid, and indicates that you
have way more latency variation than you should have
in a solid-state setup in the first place.

- and doing them on the disk, rather than closer to the
host, is doubly stupid even if you wanted to go for
an extra kicker.

Hmm? I may be wrong, but I don't think I am.

So please stop thinking that large RAM buffers "speeds
things up". Because they do nothing of the kind. Yes, they
can smooth out any non-realtime issues you have in the
GC/compaction, but you really should see it as a
"smoothing" thing, not anything else.

And there really should be no reason why SSD's should act
particularly bursty in themselves over much bigger data
sets than the size of the erase block. So you should be
able to size the buffer by roughly doing the max throughput
over an average latency cycle, and adding one erase block
worth of data to that.

With a 250MB/s thing that has 0.1 microsecond average
latencies, you shouldn't need more than a 25kB buffer. Of
course, since the erase block size is probably bigger than
that, do say 64kB or 128kB.

And then, if you support multiple outstanding commands, and
since RAM is cheap, you might decide to have separate
buffers for each outstanding command. But at that point,
you're really just wasting buffer space in order to just
not have to manage it very carefully and worry about
completion order etc.

Of course, that all depends on how smooth you can make
your GC and compaction. Maybe you can't do it entirely
incrementally. Maybe you end up having batches of a few
erase blocks.. Who knows? The point is, those latencies on
the order of a second really are the fundamental problem,
and they shouldn't exist, and buffering is not the way
to solve it.

Linus