Physical vs effective write latency

By: Doug Siebert (foo.delete@this.bar.bar), September 11, 2008 10:28 pm
Room: Moderated Discussions
Linus Torvalds (torvalds@linux-foundation.org) on 9/11/08 wrote:
---------------------------
>Doug Siebert (foo@bar.bar) on 9/11/08 wrote:
>>
>>Regular RAM and a small (pre-erased) reserved section of
>>flash along with a capacitor would be a much better
>>solution.
>
>If you have pre-erased flash, why bother with the whole
>charade to begin with?


You can't use pre-erased flash as a cache because it'll wear out, and it would need to be erased each time. And even if those two things weren't true, writing to flash, while much much faster than writing to a hard drive, is still much much slower than writing to DRAM.


>
>Guys, this is basic queuing theory. You cannot get write
>bandwidth higher than what your flash is able to absorb
>(and that includes all the GC and erase cycles necessary)
>in the steady state anyway.
>
>No amount of buffering will ever change that.
>
>Anybody who thinks that they can lower latency by just
>adding buffers is a moron, because the fact is,
>if you "lie" and tell the OS that the write has completed,
>it will just write more. Until your buffer is full, and
>you have to throttle the writes!


Look, I'm cetainly not making any ridiculous claims like saying this will increase write bandwidth over the steady state case. And of course it won't lower the underlying latency, whether it takes 100 us or 10 ms to write a block to flash. But it definitely WILL reduce the latency experienced by the end user.

You are correct that so long as there is room in the cache, and the device immediately reports the completion of the write, the OS will just write more (of course, with NCQ it is going to do that ANYWAY...) But the only reason the device would ever need to throttle writes is if the flash bandwidth is slower than the I/O channel bandwidth. With writes of 170 MB/sec in Intel's "extreme" device that will be a problem. But its not as though everytime you do writes you do many megabytes of writes. I do 100x more things that cause a few KB to be written than stuff I do that causes dozens of megabytes to be written all at once.

If Intel's SSD could sustain 300 MB/sec instead of 170 MB/sec (including any erases, GC overhead, etc.) then the 32 MB DRAM cache current generation hard drives use would be adequate to report ~1 us latency back to the OS all day long, even if those writes really take anywhere from 100 us all the way up to 100 ms(!) to write to flash (I know Intel's SSDs don't suck nearly that bad, but that's what 32 MB would permit masking based on 300 MB/sec write bandwidth) Note I'm thinking about writes on the order of 4K, if you use larger I/Os the latency is larger simply due to how long it takes the data to traverse the SATA channel, etc.

While the latency of good flash, like Intel's, is 50-100x better than a hard drive, its also 50-100x WORSE than DRAM (based on a 4K block) Which is why that DRAM cache can still help you out in masking that latency.

You are so busy worrying about good or bad implementations of GC that you are ignoring the fact that even if you had an infinite sized flash device into which you could stream your writes, you'd still have latencies in the ~100 us range with flash which could be 50-100x better if you were writing into DRAM.
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
First Dunnington benchmark resultsMichael S2008/08/19 10:54 AM
  First Dunnington benchmark resultsrwessel2008/08/19 01:42 PM
    First Dunnington benchmark resultsAaron Apink2008/08/19 05:49 PM
      First Dunnington benchmark resultsJoe Chang2008/08/19 06:28 PM
        First Dunnington benchmark resultsrwessel2008/08/21 09:49 AM
          First Dunnington benchmark resultsJoe Chang2008/08/21 03:10 PM
            First Dunnington benchmark resultsrwessel2008/08/21 06:42 PM
              First Dunnington benchmark resultsJoe Chang2008/08/21 07:12 PM
      First Dunnington benchmark resultsrwessel2008/08/21 09:45 AM
        First Dunnington benchmark resultsAaron Spink2008/08/21 01:12 PM
          First Dunnington benchmark resultsJoe Chang2008/08/21 03:15 PM
    First Dunnington benchmark resultsRichard Cownie2008/08/20 02:59 AM
      First Dunnington benchmark resultsAnders Jensen2008/08/20 03:26 AM
        +SSDAnders Jensen2008/08/20 03:30 AM
        First Dunnington benchmark resultsRichard Cownie2008/08/20 11:04 AM
          First Dunnington benchmark resultsslacker2008/08/20 12:35 PM
            First Dunnington benchmark resultsDoug Siebert2008/08/20 07:54 PM
              First Dunnington benchmark resultsRichard Cownie2008/08/20 08:58 PM
                SLC vs. MLCDavid Kanter2008/08/21 01:16 AM
                  SLC vs. MLCMatt Sayler2008/08/21 06:25 AM
                  SLC vs. MLCRichard Cownie2008/08/21 06:32 AM
                    SLC vs. MLCLinus Torvalds2008/08/21 08:39 AM
                      SLC vs. MLCMichael S2008/08/21 09:07 AM
                        SLC vs. MLCLinus Torvalds2008/08/21 09:52 AM
                          SLC vs. MLCMichael S2008/08/21 10:35 AM
                            OLTP appliance = mainframe? (NT)Potatoswatter2008/08/21 11:44 AM
                              OLTP appliance = HP NonStop?Michael S2008/08/21 12:03 PM
                                OLTP applianceJoe Chang2008/08/21 03:33 PM
                                  OLTP appliancePotatoswatter2008/08/21 03:59 PM
                            SLC vs. MLCAaron Spink2008/08/21 01:29 PM
                          SLC vs. MLCDan Downs2008/08/21 11:33 AM
                            SLC vs. MLCrwessel2008/08/21 12:45 PM
                              SLC vs. MLCDan Downs2008/08/22 08:21 AM
                            SLC vs. MLCAaron Spink2008/08/21 01:34 PM
                          SLC vs. MLC vs DRAMpgerassi2008/08/21 12:24 PM
                            SLC vs. MLC vs DRAMDavid Kanter2008/08/22 01:31 AM
                          SLC vs. MLCGroo2008/08/23 12:52 PM
                      SLC vs. MLCDoug Siebert2008/08/21 06:14 PM
                        SLC vs. MLCLinus Torvalds2008/08/22 08:05 AM
                          SLC vs. MLCDoug Siebert2008/08/22 02:27 PM
                            SLC vs. MLCEduardoS2008/08/22 06:26 PM
                            SSD Controller differentiationDavid Kanter2008/08/22 09:35 PM
                              SSD Controller differentiationDoug Siebert2008/08/22 10:34 PM
                                SSD Controller differentiation (supercaps, cost...)anon2008/08/23 10:18 AM
                                  SSD Controller differentiation (supercaps, cost...)Doug Siebert2008/08/23 10:40 AM
                            SLC vs. MLCLinus Torvalds2008/08/23 10:50 AM
                              SLC vs. MLCLinus Torvalds2008/09/08 12:03 PM
                                SLC vs. MLCMax2008/09/08 01:51 PM
                                  SLC vs. MLCHoward Chu2008/09/08 09:04 PM
                                    SLC vs. MLCMax2008/09/08 10:29 PM
                                      SLC vs. MLCHoward Chu2008/09/09 12:12 AM
                                        RAM vs SSD?Jouni Osmala2008/09/09 01:06 AM
                                          RAM vs SSD?Max2008/09/12 12:51 PM
                                            RAM vs SSD?EduardoS2008/09/12 04:27 PM
                                            Disk cache snapshottingMax2008/09/13 08:34 AM
                                              Disk cache snapshottingHoward Chu2008/09/14 09:58 PM
                                                Disk cache snapshottingMax2008/09/15 12:50 PM
                                        SLC vs. MLCLinus Torvalds2008/09/09 07:43 AM
                                          SLC vs. MLCHoward Chu2008/09/09 09:42 AM
                                            SLC vs. MLCLinus Torvalds2008/09/09 10:39 AM
                                              SLC vs. MLCMichael S2008/09/10 12:29 AM
                                                SLC vs. MLCanon2008/09/10 02:51 AM
                                                  SLC vs. MLCMichael S2008/09/10 03:09 AM
                                                    SLC vs. MLCMax2008/09/10 04:48 AM
                                                      SLC vs. MLCMichael S2008/09/10 05:52 AM
                                                        SLC vs. MLCMax2008/09/10 06:28 AM
                                                    SLC vs. MLCMatt Sayler2008/09/10 06:21 AM
                                                      SLC vs. MLCMichael S2008/09/10 09:17 AM
                                                    SLC vs. MLCanon2008/09/10 06:29 AM
                                                      SLC vs. MLCMichael S2008/09/10 09:23 AM
                                                        SLC vs. MLCMatt Sayler2008/09/10 10:45 AM
                                                SLC vs. MLCLinus Torvalds2008/09/10 07:25 AM
                                                  SLC vs. MLCMichael S2008/09/10 09:54 AM
                                                    SLC vs. MLCLinus Torvalds2008/09/10 10:31 AM
                                                      Physical vs effective write latencyMax2008/09/11 07:35 AM
                                                        Physical vs effective write latencyLinus Torvalds2008/09/11 09:06 AM
                                                          Physical vs effective write latencyLinus Torvalds2008/09/11 09:48 AM
                                                            Physical vs effective write latencyLinus Torvalds2008/09/11 11:39 AM
                                                              Physical vs effective write latencyMark Roulo2008/09/11 12:18 PM
                                                                Physical vs effective write latencyDoug Siebert2008/09/11 05:59 PM
                                                                  Physical vs effective write latencyLinus Torvalds2008/09/11 07:16 PM
                                                                    Physical vs effective write latencyDoug Siebert2008/09/11 10:28 PM
                                                        Physical vs effective write latencyMS2009/02/03 03:06 PM
                                                  SLC vs. MLC - the trick to latencyAnonymous2008/09/11 12:39 PM
                                                    SLC vs. MLC - the trick to latencyanon2008/09/11 01:17 PM
                                                      SLC vs. MLC - the trick to latencyAnonymous2008/09/11 05:25 PM
                                                        SLC vs. MLC - the trick to latencyDoug Siebert2008/09/11 05:47 PM
                                                          SLC vs. MLC - the trick to latencyrwessel2008/09/11 06:01 PM
                                                        SLC vs. MLC - the trick to latencyanon2008/09/12 12:00 AM
                                                          SLC vs. MLC - the trick to latencyAnonymous2008/09/12 08:52 PM
                                                            SLC vs. MLC - the trick to latencyanon2008/09/13 10:06 AM
                                                            SLC vs. MLC - the trick to latencyUngo2008/09/15 12:18 PM
                                                              To SSD or not? One lady's perspectiveDavid Kanter2008/09/22 01:12 AM
                                                                To SSD or not? One lady's perspectiveHoward Chu2008/09/22 04:02 AM
                                                                  To SSD or not? Real data..Linus Torvalds2008/09/22 07:33 AM
                                                                    To SSD or not? Real data..Ungo2008/09/22 12:27 PM
                                                                      4K sectorsWes Felter2008/09/22 06:03 PM
                                                                        4K sectorsDaniel2008/09/22 10:31 PM
                                                                      Reasons for >512 byte sectorsDoug Siebert2008/09/22 09:38 PM
                                                                        Reasons for >512 byte sectorsrwessel2008/09/22 10:09 PM
                                                                          Reasons for >512 byte sectorsHoward Chu2008/09/23 02:50 AM
                                                                        Reasons for >512 byte sectorsDaniel2008/09/22 10:40 PM
                                                                          Reasons for >512 byte sectorsrwessel2008/09/23 09:11 AM
                                                                            Reasons for >512 byte sectorsDaniel2008/09/23 12:10 PM
                                                                      HDD long sector size availabilityEtienne Lehnart2008/09/23 05:32 AM
                                                                        HDD long sector size availabilityrwessel2008/09/23 09:19 AM
                                                                          HDD long sector size availabilityEtienne Lehnart2008/09/23 02:17 PM
                                                                    To SSD or not? Real data..Jouni Osmala2008/09/22 11:16 PM
                                                                  To SSD or not? One lady's perspectiveWes Felter2008/09/22 11:25 AM
                                                                How should SSDs be engineered into systems?Rob Thorpe2008/09/22 02:01 PM
                                                                  How should SSDs be engineered into systems?Matt Craighead2008/09/23 06:59 PM
                                                                    How should SSDs be engineered into systems?Matt Sayler2008/09/24 04:17 AM
                                                            ATA/SCSIS, Write Flushes and Asych FilesystemsTruePath2009/01/25 04:44 AM
                                                        SLC vs. MLC - the trick to latencyMichael S2008/09/12 04:58 AM
                                                        overlapped erase and read Michael S2008/09/12 04:59 AM
                                                          overlapped erase and read David W. Hess2008/09/12 09:56 AM
                                                          overlapped erase and read Anonymous2008/09/12 08:45 PM
                                                            overlapped erase and read Jouni Osmala2008/09/12 11:56 PM
                                                            overlapped erase and read Michael S2008/09/13 11:29 AM
                                                            overlapped erase and read Michael S2008/09/13 12:09 PM
                                                              overlapped erase and read Linus Torvalds2008/09/13 02:05 PM
                                                    SLC vs. MLC - the trick to latencyDoug Siebert2008/09/11 05:31 PM
                                SLC vs. MLCEduardoS2008/09/08 02:07 PM
                                  SLC vs. MLCLinus Torvalds2008/09/08 02:30 PM
                                    SLC vs. MLCEduardoS2008/09/08 04:01 PM
                                      SSD and RAIDJoe Chang2008/09/08 07:42 PM
                                        SSD and RAIDDoug Siebert2008/09/08 09:46 PM
                                          SSD and RAIDAaron Spink2008/09/09 04:27 PM
                                            SSD and RAIDGroo2008/09/10 01:02 PM
                              SLC vs. MLCJoern Engel2009/01/06 10:22 AM
                                SLC vs. MLCLinus Torvalds2009/01/06 02:04 PM
                                  SLC vs. MLCJoern Engel2009/01/06 03:24 PM
                                    SLC vs. MLCrwessel2009/01/06 04:47 PM
                                      SLC vs. MLCanonymous2009/01/06 05:17 PM
                                        SLC vs. MLCrwessel2009/01/06 05:58 PM
                                      SLC vs. MLCJoern Engel2009/01/07 12:35 AM
                                    SLC vs. MLCLinus Torvalds2009/01/06 05:45 PM
                                      SLC vs. MLCrwessel2009/01/06 06:09 PM
                                        SLC vs. MLCLinus Torvalds2009/01/06 07:47 PM
                                      SLC vs. MLCJoern Engel2009/01/07 12:26 AM
                                    SLC vs. MLCanon2009/01/06 08:23 PM
                                      SLC vs. MLCJoern Engel2009/01/07 12:52 AM
                                        SLC vs. MLCanon2009/01/07 02:34 AM
                                  SLC vs. MLCIntelUser20002009/01/07 07:43 AM
                                    SLC vs. MLCLinus Torvalds2009/01/07 10:28 AM
                                      drop data filesystem semanticDoug Siebert2009/01/09 12:21 PM
                                      FTL and FSiz2009/01/09 07:49 PM
                                        FTL and FSLinus Torvalds2009/01/09 09:53 PM
                                          FTL and FSiz2009/01/10 02:09 AM
                                            FTL and FSMichael S2009/01/10 03:19 PM
                                              compiling large programsiz2009/01/10 05:51 PM
                                                compiling large programsLinus Torvalds2009/01/10 07:58 PM
                                                  compiling large programspeter2009/01/11 05:30 AM
                                                  compiling large programsAndi Kleen2009/01/11 01:03 PM
                                                  The File AbstractionTruePath2009/01/25 06:45 AM
                                                    The File AbstractionHoward Chu2009/01/25 01:49 PM
                                                      The File AbstractionLinus Torvalds2009/01/26 09:23 AM
                                                        The File AbstractionMichael S2009/01/26 01:39 PM
                                                          The File AbstractionLinus Torvalds2009/01/26 02:31 PM
                                                            The File AbstractionDean Kent2009/01/26 03:06 PM
                                                              The File AbstractionLinus Torvalds2009/01/26 04:29 PM
                                                                The File AbstractionMark Christiansen2009/01/27 09:24 AM
                                                                  The File AbstractionMark Christiansen2009/01/27 10:14 AM
                                                                  The File AbstractionLinus Torvalds2009/01/27 10:15 AM
                                                                    The File Abstractionslacker2009/01/27 11:20 AM
                                                                      The File AbstractionLinus Torvalds2009/01/27 01:16 PM
                                                                        Attributes All The Way DownMark Christiansen2009/01/27 02:17 PM
                                                                        The File Abstractionslacker2009/01/27 05:25 PM
                                                                          The File AbstractionLinus Torvalds2009/01/28 08:17 AM
                                                                            The File Abstraction: API thoughtsCarlie Coats2009/01/28 09:35 AM
                                                                            The File Abstractionslacker2009/01/28 10:09 AM
                                                                              The File AbstractionLinus Torvalds2009/01/28 01:44 PM
                                                                                Programs already 'hide' their metadata in the bytestream, unbeknownst to usersanon2009/01/28 09:28 PM
                                                                                The File Abstractionslacker2009/01/29 10:39 AM
                                                                                  The File AbstractionLinus Torvalds2009/01/29 11:08 AM
                                                                                    The File AbstractionDean Kent2009/01/29 11:49 AM
                                                                                      The File AbstractionHoward Chu2009/01/29 02:58 PM
                                                                                        The File Abstractionrwessel2009/01/29 04:23 PM
                                                                                    Extended Attributes in Actionslacker2009/01/29 03:05 PM
                                                                                      Extended Attributes in Actionstubar2009/01/29 04:49 PM
                                                                                      Extended Attributes in ActionLinus Torvalds2009/01/29 05:15 PM
                                                                                        Like Duhanon2009/01/29 07:42 PM
                                                                                          Like Duhanon2009/01/29 09:15 PM
                                                                                            Like Duhanon2009/02/01 07:18 PM
                                                                                              Double Duh.Anonymous2009/02/01 10:58 PM
                                                                                                Double Duh.anon2009/02/02 02:08 AM
                                                                                                  Double Duh.Anonymous2009/02/02 05:11 PM
                                                                                                    Double Duh.anon2009/02/02 07:33 PM
                                                                                              Like DuhDavid Kanter2009/02/01 11:05 PM
                                                                                                Like Duhpeter2009/02/01 11:55 PM
                                                                                                Like Duhanon2009/02/02 01:55 AM
                                                                                                Xattrs, Solar power, regulation and politicsRob Thorpe2009/02/02 04:36 AM
                                                                                                  Terminology seems too fuzzy to mehobold2009/02/02 06:14 AM
                                                                                                    Terminology seems too fuzzy to merwessel2009/02/02 12:33 PM
                                                                                                      good summaryMichael S2009/02/03 02:41 AM
                                                                                                        good summaryMark Christiansen2009/02/03 09:57 AM
                                                                                                          good summaryHoward Chu2009/02/03 10:21 AM
                                                                                                            good summaryMark Christiansen2009/02/03 11:18 AM
                                                                                                              good summaryHoward Chu2009/02/03 12:00 PM
                                                                                                                good summaryMark Christiansen2009/02/03 12:36 PM
                                                                                                          good summaryRagingDragon2009/02/03 10:39 PM
                                                                                                            good summaryrwessel2009/02/03 11:03 PM
                                                                                                              good summaryRagingDragon2009/02/03 11:46 PM
                                                                                                      Terminology seems too fuzzy to meslacker2009/02/04 05:06 PM
                                                                                                        Terminology seems too fuzzy to meMichael S2009/02/05 01:05 AM
                                                                                                          Terminology seems too fuzzy to meUngo2009/02/05 01:15 PM
                                                                                                          Terminology seems too fuzzy to meslacker2009/02/05 02:19 PM
                                                                                                            Terminology seems too fuzzy to meHoward Chu2009/02/05 04:44 PM
                                                                                          Like Duhiz2009/01/30 02:03 AM
                                                                                          EAs (security labels) hosed me badlyanon2009/01/30 09:48 PM
                                                                                        Extended Attributes in ActionRagingDragon2009/01/29 09:31 PM
                                                                                      Extended Attributes in Actionanonymous2009/01/29 08:13 PM
                                                                                      Extended Attributes in ActionHoward Chu2009/01/29 09:38 PM
                                                                                        Extended Attributes in Actionslacker2009/01/30 11:24 AM
                                                                                          Extended Attributes in Actionanon2009/01/30 05:50 PM
                                                                                      Extended Attributes in ActionEtienne Lehnart2009/01/30 12:22 AM
                                                                                      Extended Attributes in ActionRob Thorpe2009/01/30 12:39 PM
                                                                                        Extended Attributes in Actionslacker2009/01/30 01:16 PM
                                                                                          Extended Attributes in Actionanon2009/01/30 06:03 PM
                                                                                          Extended Attributes in ActionHoward Chu2009/01/30 11:22 PM
                                                                                            Extended Attributes in Actionrwessel2009/01/31 12:08 AM
                                                                                              Extended Attributes in Actionanonymous2009/01/31 12:22 AM
                                                                                                Extended Attributes in Actionrwessel2009/01/31 12:56 AM
                                                                                                  ScalingDean Kent2009/01/31 09:04 AM
                                                                                                    ScalingRob Thorpe2009/02/02 02:39 AM
                                                                                                      Scalingrwessel2009/02/02 11:41 AM
                                                                                                        ScalingHoward Chu2009/02/02 12:30 PM
                                                                                                          ScalingDean Kent2009/02/02 02:27 PM
                                                                                                            ScalingRob Thorpe2009/02/03 05:08 AM
                                                                                                              ScalingDean Kent2009/02/03 07:38 AM
                                                                                                              Scalingrwessel2009/02/03 02:34 PM
                                                                                                                ScalingRagingDragon2009/02/03 10:46 PM
                                                                                                    in defense of software that does not scaleMatt Sayler2009/02/03 11:27 AM
                                                                                                      in defense of software that does not scaleHoward Chu2009/02/03 12:03 PM
                                                                                                        in defense of software that does not scaleMatt Sayler2009/02/03 12:17 PM
                                                                                                        in defense of software that does not scaleRagingDragon2009/02/03 11:00 PM
                                                                                                          in defense of software that does not scaleMichael S2009/02/04 06:46 AM
                                                                                                            in defense of software that does not scaleRagingDragon2009/02/04 09:33 PM
                                                                                                      in defense of software that does not scaleDean Kent2009/02/03 12:17 PM
                                                                                                        in defense of software that does not scaleMatt Sayler2009/02/03 12:24 PM
                                                                                                        in defense of software that does not scaleVincent Diepeveen2009/02/04 10:43 AM
                                                                                                      in defense of software that does not scalerwessel2009/02/03 02:44 PM
                                                                                                        in defense of software that does not scaleanon2009/02/04 02:35 AM
                                                                                                          in defense of software that does not scaleCarlie Coats2009/02/04 05:24 AM
                                                                                                      Scaling with time vs. scaling from the beginning.mpx2009/02/05 01:57 AM
                                                                                                  Extended Attributes in ActionMichael S2009/01/31 10:33 AM
                                                                                                  Extended Attributes in Actionanon2009/01/31 10:37 PM
                                                                                          Extended Attributes in ActionJasonB2009/01/31 08:11 AM
                                                                                            Extended Attributes in ActionHoward Chu2009/01/31 11:43 AM
                                                                                              Extended Attributes in ActionJasonB2009/01/31 04:37 PM
                                                                                                Extended Attributes in ActionHoward Chu2009/02/02 02:42 PM
                                                                                                  Extended Attributes in ActionHoward Chu2009/02/02 02:44 PM
                                                                    The File AbstractionRob Thorpe2009/01/27 11:20 AM
                                                              The File AbstractionHoward Chu2009/01/27 12:28 AM
                                                                The File AbstractionMichael S2009/01/27 03:00 AM
                                                                The File AbstractionDean Kent2009/01/27 08:30 AM
                                                        The File AbstractionAndi Kleen2009/01/27 02:05 AM
                                  SLC vs. MLCMichel2009/01/12 06:54 PM
                                    SLC vs. MLCLinus Torvalds2009/01/12 07:38 PM
                                      SLC vs. MLCrwessel2009/01/13 12:52 AM
                                      SLC vs. MLCUngo2009/01/13 03:04 PM
                                        SLC vs. MLCWes Felter2009/01/13 05:42 PM
                                      SLC vs. MLCTruePath2009/01/25 05:05 AM
                    SLC vs. MLCUngo2008/08/21 12:54 PM
                    SLC vs. MLCAaron Spink2008/08/21 01:20 PM
                  MLC vs. SLCMichael S2008/08/21 08:57 AM
                First Dunnington benchmark resultsrwessel2008/08/21 10:40 AM
              First Dunnington benchmark resultsAaron Spink2008/08/21 03:18 AM
      First Dunnington benchmark resultsEtienne Lehnart2008/08/20 04:38 AM
  Will x86 dominate big iron?Tom W2008/08/19 10:10 PM
    Will x86 dominate big iron?Jesper Frimann2008/08/20 12:28 AM
      Will x86 dominate big iron?Tom W2008/08/20 03:42 PM
        Will x86 dominate big iron?David Kanter2008/08/21 01:13 AM
          Will x86 dominate big iron?Joe Chang2008/08/21 06:54 AM
            Will x86 dominate big iron?asdf2008/08/22 01:18 PM
              Will x86 dominate big iron?Dean Kent2008/08/22 07:54 PM
        Will x86 dominate big iron?Jesper Frimann2008/08/22 09:48 AM
          Will x86 dominate big iron?Tom W2008/08/24 01:06 AM
            Will x86 dominate big iron?Michael S2008/08/24 04:19 AM
              Will x86 dominate big iron?Dean Kent2008/08/24 09:30 AM
                Will x86 dominate big iron?Paul2008/08/24 11:16 AM
                  Will x86 dominate big iron?Dean Kent2008/08/24 12:37 PM
                Will x86 dominate big iron?Michael S2008/08/25 12:53 AM
        Will x86 dominate big iron?someone2008/08/22 10:19 AM
          Will x86 dominate big iron?aaron spink2008/08/23 02:56 AM
            Will x86 dominate big iron?Michael S2008/08/23 09:58 AM
              Will x86 dominate big iron?someone2008/08/23 01:51 PM
            Will x86 dominate big iron?someone2008/08/23 01:55 PM
              Will x86 dominate big iron?Aaron Spink2008/08/23 04:52 PM
                Will x86 dominate big iron?anonymous2008/08/23 05:28 PM
                  Will x86 dominate big iron?Dean Kent2008/08/23 06:12 PM
                    Off road and topicEduardoS2008/08/23 06:28 PM
                  Will x86 dominate big iron?someone2008/08/23 06:26 PM
                    Will x86 dominate big iron?Dean Kent2008/08/23 09:40 PM
                    Will x86 dominate big iron?anonymous2008/08/24 01:46 AM
                  Off road and topicDavid W. Hess2008/08/24 03:24 AM
                    Off road and topicAaron Spink2008/08/24 04:14 AM
  Beckton vs. DunningtonMr. Camel2008/08/22 06:30 AM
    Beckton vs. Dunningtonjokerman2008/08/22 12:12 PM
    Beckton vs. DunningtonMr. Camel2009/05/29 10:16 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell green?