Linus Torvalds (torvalds@linux-foundation.org) on 9/11/08 wrote:
---------------------------
>Doug Siebert (foo@bar.bar) on 9/11/08 wrote:
>>
>>Regular RAM and a small (pre-erased) reserved section of
>>flash along with a capacitor would be a much better
>>solution.
>
>If you have pre-erased flash, why bother with the whole
>charade to begin with?


You can't use pre-erased flash as a cache because it'll wear out, and it would need to be erased each time. And even if those two things weren't true, writing to flash, while much much faster than writing to a hard drive, is still much much slower than writing to DRAM.


>
>Guys, this is basic queuing theory. You cannot get write
>bandwidth higher than what your flash is able to absorb
>(and that includes all the GC and erase cycles necessary)
>in the steady state anyway.
>
>No amount of buffering will ever change that.
>
>Anybody who thinks that they can lower latency by just
>adding buffers is a moron, because the fact is,
>if you "lie" and tell the OS that the write has completed,
>it will just write more. Until your buffer is full, and
>you have to throttle the writes!


Look, I'm cetainly not making any ridiculous claims like saying this will increase write bandwidth over the steady state case. And of course it won't lower the underlying latency, whether it takes 100 us or 10 ms to write a block to flash. But it definitely WILL reduce the latency experienced by the end user.

You are correct that so long as there is room in the cache, and the device immediately reports the completion of the write, the OS will just write more (of course, with NCQ it is going to do that ANYWAY...) But the only reason the device would ever need to throttle writes is if the flash bandwidth is slower than the I/O channel bandwidth. With writes of 170 MB/sec in Intel's "extreme" device that will be a problem. But its not as though everytime you do writes you do many megabytes of writes. I do 100x more things that cause a few KB to be written than stuff I do that causes dozens of megabytes to be written all at once.

If Intel's SSD could sustain 300 MB/sec instead of 170 MB/sec (including any erases, GC overhead, etc.) then the 32 MB DRAM cache current generation hard drives use would be adequate to report ~1 us latency back to the OS all day long, even if those writes really take anywhere from 100 us all the way up to 100 ms(!) to write to flash (I know Intel's SSDs don't suck nearly that bad, but that's what 32 MB would permit masking based on 300 MB/sec write bandwidth) Note I'm thinking about writes on the order of 4K, if you use larger I/Os the latency is larger simply due to how long it takes the data to traverse the SATA channel, etc.

While the latency of good flash, like Intel's, is 50-100x better than a hard drive, its also 50-100x WORSE than DRAM (based on a 4K block) Which is why that DRAM cache can still help you out in masking that latency.

You are so busy worrying about good or bad implementations of GC that you are ignoring the fact that even if you had an infinite sized flash device into which you could stream your writes, you'd still have latencies in the ~100 us range with flash which could be 50-100x better if you were writing into DRAM.