By: rwessel (robertwessel.delete@this.yahoo.com), December 8, 2014 12:52 pm
Room: Moderated Discussions
Brett (ggtgp.delete@this.yahoo.com) on December 8, 2014 10:00 am wrote:
> Eric Bron (eric.bron.delete@this.zvisuel.privatefortest.com) on December 8, 2014 4:35 am wrote:
> > > have huge working sets which would take too much memory, energy, and loading time to access.
> >
> > even without resorting to new technologies, SSD are vey fast already, something which need
> > much attention for out of core processing is the file formats, legacy formats and many new
> > formats (!) are based on serialization of the whole file which goes against parallelization
>
> A SSD has one set of pins you plug the device into, so ultimately the interface is serial.
> The big mistake is not preallocating your disk blocks, a simple byte stream will have to update the file
> table for each block added. This doubles the number of blocks written to and almost all those file table
> blocks will need to be erased, which leads to a slow garbage collection wear leveling pass.
Unless you specify immediate writes, better file systems do not need to commit the block indexes/pointers to disk for a file being sequentially written until the data is actually written to disk, or even until the file is actually closed. And considerable amounts of data can be buffered.
> Eric Bron (eric.bron.delete@this.zvisuel.privatefortest.com) on December 8, 2014 4:35 am wrote:
> > > have huge working sets which would take too much memory, energy, and loading time to access.
> >
> > even without resorting to new technologies, SSD are vey fast already, something which need
> > much attention for out of core processing is the file formats, legacy formats and many new
> > formats (!) are based on serialization of the whole file which goes against parallelization
>
> A SSD has one set of pins you plug the device into, so ultimately the interface is serial.
> The big mistake is not preallocating your disk blocks, a simple byte stream will have to update the file
> table for each block added. This doubles the number of blocks written to and almost all those file table
> blocks will need to be erased, which leads to a slow garbage collection wear leveling pass.
Unless you specify immediate writes, better file systems do not need to commit the block indexes/pointers to disk for a file being sequentially written until the data is actually written to disk, or even until the file is actually closed. And considerable amounts of data can be buffered.
