Physical vs effective write latency

By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), September 11, 2008 9:06 am
Room: Moderated Discussions
Max (max@a.com) on 9/11/08 wrote:
>
>One way that a flash drive could have a lower effective
>write latency than its physical write latency is if writes
>were interruptable.
>
>Say the drive begins a slow write, then gets a read
>request. It pauses the write, does the read, then restarts
>the write.

For all I know, they may already do that. That's what having
multiple commands in flight allows (ATA calls it NCQ, aka
"Native Command Queuing"). However, you missed the point:
it doesn't really help all that much.

The reason it doesn't help all that much is that

- the latency if the individual write matters.

Even if you let other, shorter-latency, operations go
first, the latency of the write is still bad (and in
fact quite possibly got even worse).

NCQ improves the IO throughput of the device (not just
on SSD's, of course - it allows rotational media to
schedule the IO's optimally), but it also tends to make
the worst-case latencies worse even as the
average latency may improve.

And when talking various real-time loads (and the very
important issue of "interactivity" is one such thing),
worst-case latency can sometimes matter a whole lot more
than average latency.

- as mentioned, the queue depth is quite limited.

The command queue is often 16 or 32 entries, and if you
are doing writes, you're often doing them in bunches.
That's exactly because most writes are bufferable, and
when you do synchronous writes, they are almost always
not "single" writes (ie you may be doing a sync on a
file, but that will be all the file data and the meta-
data that goes with it, and then you'd wait synchronously
for that bunch of IO's to all have completed).

IOW, you'd expect that when doing writes, the queue is
often filled. Yes, you could limit the number of queued
writes, but since writes are generally where queueing
helps the most (since they are the ones that are
the most independent), that's going to hurt throughput,
and is pretty complicated anyway.

- Especially for something like a GC event, it's not that
it's a single write command that would be blocked,
it's that all of them end up blocking on GC when you
do end up having a blocking GC happen in the first place.

So if you've filled the queue with writes, they're all
going to be there, and none of them is going to succeed
early, and that read won't get through simply because
the queue is full.

Anyway, regardless of any of the queuing issues - even if
you let reads complete first - it's still going to cause
problems for those synchronous writes. And those synchronous
writes are not at all as uncommon as people think.

They happen for a lot of filesystems for just metadata
updates (ie file create/delete events), and they always
happen when the app asked for them (ie fdatasync() in a DB,
or just fsync() for a mail reader etc).

And yes, many of them do happen for simple everyday apps.
Browsers, mail readers, things like that. Apps where ten
milliseconds is not very noticeable, but half a second is
very noticeable and irritating indeed.

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
First Dunnington benchmark resultsMichael S2008/08/19 10:54 AM
  First Dunnington benchmark resultsrwessel2008/08/19 01:42 PM
    First Dunnington benchmark resultsAaron Apink2008/08/19 05:49 PM
      First Dunnington benchmark resultsJoe Chang2008/08/19 06:28 PM
        First Dunnington benchmark resultsrwessel2008/08/21 09:49 AM
          First Dunnington benchmark resultsJoe Chang2008/08/21 03:10 PM
            First Dunnington benchmark resultsrwessel2008/08/21 06:42 PM
              First Dunnington benchmark resultsJoe Chang2008/08/21 07:12 PM
      First Dunnington benchmark resultsrwessel2008/08/21 09:45 AM
        First Dunnington benchmark resultsAaron Spink2008/08/21 01:12 PM
          First Dunnington benchmark resultsJoe Chang2008/08/21 03:15 PM
    First Dunnington benchmark resultsRichard Cownie2008/08/20 02:59 AM
      First Dunnington benchmark resultsAnders Jensen2008/08/20 03:26 AM
        +SSDAnders Jensen2008/08/20 03:30 AM
        First Dunnington benchmark resultsRichard Cownie2008/08/20 11:04 AM
          First Dunnington benchmark resultsslacker2008/08/20 12:35 PM
            First Dunnington benchmark resultsDoug Siebert2008/08/20 07:54 PM
              First Dunnington benchmark resultsRichard Cownie2008/08/20 08:58 PM
                SLC vs. MLCDavid Kanter2008/08/21 01:16 AM
                  SLC vs. MLCMatt Sayler2008/08/21 06:25 AM
                  SLC vs. MLCRichard Cownie2008/08/21 06:32 AM
                    SLC vs. MLCLinus Torvalds2008/08/21 08:39 AM
                      SLC vs. MLCMichael S2008/08/21 09:07 AM
                        SLC vs. MLCLinus Torvalds2008/08/21 09:52 AM
                          SLC vs. MLCMichael S2008/08/21 10:35 AM
                            OLTP appliance = mainframe? (NT)Potatoswatter2008/08/21 11:44 AM
                              OLTP appliance = HP NonStop?Michael S2008/08/21 12:03 PM
                                OLTP applianceJoe Chang2008/08/21 03:33 PM
                                  OLTP appliancePotatoswatter2008/08/21 03:59 PM
                            SLC vs. MLCAaron Spink2008/08/21 01:29 PM
                          SLC vs. MLCDan Downs2008/08/21 11:33 AM
                            SLC vs. MLCrwessel2008/08/21 12:45 PM
                              SLC vs. MLCDan Downs2008/08/22 08:21 AM
                            SLC vs. MLCAaron Spink2008/08/21 01:34 PM
                          SLC vs. MLC vs DRAMpgerassi2008/08/21 12:24 PM
                            SLC vs. MLC vs DRAMDavid Kanter2008/08/22 01:31 AM
                          SLC vs. MLCGroo2008/08/23 12:52 PM
                      SLC vs. MLCDoug Siebert2008/08/21 06:14 PM
                        SLC vs. MLCLinus Torvalds2008/08/22 08:05 AM
                          SLC vs. MLCDoug Siebert2008/08/22 02:27 PM
                            SLC vs. MLCEduardoS2008/08/22 06:26 PM
                            SSD Controller differentiationDavid Kanter2008/08/22 09:35 PM
                              SSD Controller differentiationDoug Siebert2008/08/22 10:34 PM
                                SSD Controller differentiation (supercaps, cost...)anon2008/08/23 10:18 AM
                                  SSD Controller differentiation (supercaps, cost...)Doug Siebert2008/08/23 10:40 AM
                            SLC vs. MLCLinus Torvalds2008/08/23 10:50 AM
                              SLC vs. MLCLinus Torvalds2008/09/08 12:03 PM
                                SLC vs. MLCMax2008/09/08 01:51 PM
                                  SLC vs. MLCHoward Chu2008/09/08 09:04 PM
                                    SLC vs. MLCMax2008/09/08 10:29 PM
                                      SLC vs. MLCHoward Chu2008/09/09 12:12 AM
                                        RAM vs SSD?Jouni Osmala2008/09/09 01:06 AM
                                          RAM vs SSD?Max2008/09/12 12:51 PM
                                            RAM vs SSD?EduardoS2008/09/12 04:27 PM
                                            Disk cache snapshottingMax2008/09/13 08:34 AM
                                              Disk cache snapshottingHoward Chu2008/09/14 09:58 PM
                                                Disk cache snapshottingMax2008/09/15 12:50 PM
                                        SLC vs. MLCLinus Torvalds2008/09/09 07:43 AM
                                          SLC vs. MLCHoward Chu2008/09/09 09:42 AM
                                            SLC vs. MLCLinus Torvalds2008/09/09 10:39 AM
                                              SLC vs. MLCMichael S2008/09/10 12:29 AM
                                                SLC vs. MLCanon2008/09/10 02:51 AM
                                                  SLC vs. MLCMichael S2008/09/10 03:09 AM
                                                    SLC vs. MLCMax2008/09/10 04:48 AM
                                                      SLC vs. MLCMichael S2008/09/10 05:52 AM
                                                        SLC vs. MLCMax2008/09/10 06:28 AM
                                                    SLC vs. MLCMatt Sayler2008/09/10 06:21 AM
                                                      SLC vs. MLCMichael S2008/09/10 09:17 AM
                                                    SLC vs. MLCanon2008/09/10 06:29 AM
                                                      SLC vs. MLCMichael S2008/09/10 09:23 AM
                                                        SLC vs. MLCMatt Sayler2008/09/10 10:45 AM
                                                SLC vs. MLCLinus Torvalds2008/09/10 07:25 AM
                                                  SLC vs. MLCMichael S2008/09/10 09:54 AM
                                                    SLC vs. MLCLinus Torvalds2008/09/10 10:31 AM
                                                      Physical vs effective write latencyMax2008/09/11 07:35 AM
                                                        Physical vs effective write latencyLinus Torvalds2008/09/11 09:06 AM
                                                          Physical vs effective write latencyLinus Torvalds2008/09/11 09:48 AM
                                                            Physical vs effective write latencyLinus Torvalds2008/09/11 11:39 AM
                                                              Physical vs effective write latencyMark Roulo2008/09/11 12:18 PM
                                                                Physical vs effective write latencyDoug Siebert2008/09/11 05:59 PM
                                                                  Physical vs effective write latencyLinus Torvalds2008/09/11 07:16 PM
                                                                    Physical vs effective write latencyDoug Siebert2008/09/11 10:28 PM
                                                        Physical vs effective write latencyMS2009/02/03 03:06 PM
                                                  SLC vs. MLC - the trick to latencyAnonymous2008/09/11 12:39 PM
                                                    SLC vs. MLC - the trick to latencyanon2008/09/11 01:17 PM
                                                      SLC vs. MLC - the trick to latencyAnonymous2008/09/11 05:25 PM
                                                        SLC vs. MLC - the trick to latencyDoug Siebert2008/09/11 05:47 PM
                                                          SLC vs. MLC - the trick to latencyrwessel2008/09/11 06:01 PM
                                                        SLC vs. MLC - the trick to latencyanon2008/09/12 12:00 AM
                                                          SLC vs. MLC - the trick to latencyAnonymous2008/09/12 08:52 PM
                                                            SLC vs. MLC - the trick to latencyanon2008/09/13 10:06 AM
                                                            SLC vs. MLC - the trick to latencyUngo2008/09/15 12:18 PM
                                                              To SSD or not? One lady's perspectiveDavid Kanter2008/09/22 01:12 AM
                                                                To SSD or not? One lady's perspectiveHoward Chu2008/09/22 04:02 AM
                                                                  To SSD or not? Real data..Linus Torvalds2008/09/22 07:33 AM
                                                                    To SSD or not? Real data..Ungo2008/09/22 12:27 PM
                                                                      4K sectorsWes Felter2008/09/22 06:03 PM
                                                                        4K sectorsDaniel2008/09/22 10:31 PM
                                                                      Reasons for >512 byte sectorsDoug Siebert2008/09/22 09:38 PM
                                                                        Reasons for >512 byte sectorsrwessel2008/09/22 10:09 PM
                                                                          Reasons for >512 byte sectorsHoward Chu2008/09/23 02:50 AM
                                                                        Reasons for >512 byte sectorsDaniel2008/09/22 10:40 PM
                                                                          Reasons for >512 byte sectorsrwessel2008/09/23 09:11 AM
                                                                            Reasons for >512 byte sectorsDaniel2008/09/23 12:10 PM
                                                                      HDD long sector size availabilityEtienne Lehnart2008/09/23 05:32 AM
                                                                        HDD long sector size availabilityrwessel2008/09/23 09:19 AM
                                                                          HDD long sector size availabilityEtienne Lehnart2008/09/23 02:17 PM
                                                                    To SSD or not? Real data..Jouni Osmala2008/09/22 11:16 PM
                                                                  To SSD or not? One lady's perspectiveWes Felter2008/09/22 11:25 AM
                                                                How should SSDs be engineered into systems?Rob Thorpe2008/09/22 02:01 PM
                                                                  How should SSDs be engineered into systems?Matt Craighead2008/09/23 06:59 PM
                                                                    How should SSDs be engineered into systems?Matt Sayler2008/09/24 04:17 AM
                                                            ATA/SCSIS, Write Flushes and Asych FilesystemsTruePath2009/01/25 04:44 AM
                                                        SLC vs. MLC - the trick to latencyMichael S2008/09/12 04:58 AM
                                                        overlapped erase and read Michael S2008/09/12 04:59 AM
                                                          overlapped erase and read David W. Hess2008/09/12 09:56 AM
                                                          overlapped erase and read Anonymous2008/09/12 08:45 PM
                                                            overlapped erase and read Jouni Osmala2008/09/12 11:56 PM
                                                            overlapped erase and read Michael S2008/09/13 11:29 AM
                                                            overlapped erase and read Michael S2008/09/13 12:09 PM
                                                              overlapped erase and read Linus Torvalds2008/09/13 02:05 PM
                                                    SLC vs. MLC - the trick to latencyDoug Siebert2008/09/11 05:31 PM
                                SLC vs. MLCEduardoS2008/09/08 02:07 PM
                                  SLC vs. MLCLinus Torvalds2008/09/08 02:30 PM
                                    SLC vs. MLCEduardoS2008/09/08 04:01 PM
                                      SSD and RAIDJoe Chang2008/09/08 07:42 PM
                                        SSD and RAIDDoug Siebert2008/09/08 09:46 PM
                                          SSD and RAIDAaron Spink2008/09/09 04:27 PM
                                            SSD and RAIDGroo2008/09/10 01:02 PM
                              SLC vs. MLCJoern Engel2009/01/06 10:22 AM
                                SLC vs. MLCLinus Torvalds2009/01/06 02:04 PM
                                  SLC vs. MLCJoern Engel2009/01/06 03:24 PM
                                    SLC vs. MLCrwessel2009/01/06 04:47 PM
                                      SLC vs. MLCanonymous2009/01/06 05:17 PM
                                        SLC vs. MLCrwessel2009/01/06 05:58 PM
                                      SLC vs. MLCJoern Engel2009/01/07 12:35 AM
                                    SLC vs. MLCLinus Torvalds2009/01/06 05:45 PM
                                      SLC vs. MLCrwessel2009/01/06 06:09 PM
                                        SLC vs. MLCLinus Torvalds2009/01/06 07:47 PM
                                      SLC vs. MLCJoern Engel2009/01/07 12:26 AM
                                    SLC vs. MLCanon2009/01/06 08:23 PM
                                      SLC vs. MLCJoern Engel2009/01/07 12:52 AM
                                        SLC vs. MLCanon2009/01/07 02:34 AM
                                  SLC vs. MLCIntelUser20002009/01/07 07:43 AM
                                    SLC vs. MLCLinus Torvalds2009/01/07 10:28 AM
                                      drop data filesystem semanticDoug Siebert2009/01/09 12:21 PM
                                      FTL and FSiz2009/01/09 07:49 PM
                                        FTL and FSLinus Torvalds2009/01/09 09:53 PM
                                          FTL and FSiz2009/01/10 02:09 AM
                                            FTL and FSMichael S2009/01/10 03:19 PM
                                              compiling large programsiz2009/01/10 05:51 PM
                                                compiling large programsLinus Torvalds2009/01/10 07:58 PM
                                                  compiling large programspeter2009/01/11 05:30 AM
                                                  compiling large programsAndi Kleen2009/01/11 01:03 PM
                                                  The File AbstractionTruePath2009/01/25 06:45 AM
                                                    The File AbstractionHoward Chu2009/01/25 01:49 PM
                                                      The File AbstractionLinus Torvalds2009/01/26 09:23 AM
                                                        The File AbstractionMichael S2009/01/26 01:39 PM
                                                          The File AbstractionLinus Torvalds2009/01/26 02:31 PM
                                                            The File AbstractionDean Kent2009/01/26 03:06 PM
                                                              The File AbstractionLinus Torvalds2009/01/26 04:29 PM
                                                                The File AbstractionMark Christiansen2009/01/27 09:24 AM
                                                                  The File AbstractionMark Christiansen2009/01/27 10:14 AM
                                                                  The File AbstractionLinus Torvalds2009/01/27 10:15 AM
                                                                    The File Abstractionslacker2009/01/27 11:20 AM
                                                                      The File AbstractionLinus Torvalds2009/01/27 01:16 PM
                                                                        Attributes All The Way DownMark Christiansen2009/01/27 02:17 PM
                                                                        The File Abstractionslacker2009/01/27 05:25 PM
                                                                          The File AbstractionLinus Torvalds2009/01/28 08:17 AM
                                                                            The File Abstraction: API thoughtsCarlie Coats2009/01/28 09:35 AM
                                                                            The File Abstractionslacker2009/01/28 10:09 AM
                                                                              The File AbstractionLinus Torvalds2009/01/28 01:44 PM
                                                                                Programs already 'hide' their metadata in the bytestream, unbeknownst to usersanon2009/01/28 09:28 PM
                                                                                The File Abstractionslacker2009/01/29 10:39 AM
                                                                                  The File AbstractionLinus Torvalds2009/01/29 11:08 AM
                                                                                    The File AbstractionDean Kent2009/01/29 11:49 AM
                                                                                      The File AbstractionHoward Chu2009/01/29 02:58 PM
                                                                                        The File Abstractionrwessel2009/01/29 04:23 PM
                                                                                    Extended Attributes in Actionslacker2009/01/29 03:05 PM
                                                                                      Extended Attributes in Actionstubar2009/01/29 04:49 PM
                                                                                      Extended Attributes in ActionLinus Torvalds2009/01/29 05:15 PM
                                                                                        Like Duhanon2009/01/29 07:42 PM
                                                                                          Like Duhanon2009/01/29 09:15 PM
                                                                                            Like Duhanon2009/02/01 07:18 PM
                                                                                              Double Duh.Anonymous2009/02/01 10:58 PM
                                                                                                Double Duh.anon2009/02/02 02:08 AM
                                                                                                  Double Duh.Anonymous2009/02/02 05:11 PM
                                                                                                    Double Duh.anon2009/02/02 07:33 PM
                                                                                              Like DuhDavid Kanter2009/02/01 11:05 PM
                                                                                                Like Duhpeter2009/02/01 11:55 PM
                                                                                                Like Duhanon2009/02/02 01:55 AM
                                                                                                Xattrs, Solar power, regulation and politicsRob Thorpe2009/02/02 04:36 AM
                                                                                                  Terminology seems too fuzzy to mehobold2009/02/02 06:14 AM
                                                                                                    Terminology seems too fuzzy to merwessel2009/02/02 12:33 PM
                                                                                                      good summaryMichael S2009/02/03 02:41 AM
                                                                                                        good summaryMark Christiansen2009/02/03 09:57 AM
                                                                                                          good summaryHoward Chu2009/02/03 10:21 AM
                                                                                                            good summaryMark Christiansen2009/02/03 11:18 AM
                                                                                                              good summaryHoward Chu2009/02/03 12:00 PM
                                                                                                                good summaryMark Christiansen2009/02/03 12:36 PM
                                                                                                          good summaryRagingDragon2009/02/03 10:39 PM
                                                                                                            good summaryrwessel2009/02/03 11:03 PM
                                                                                                              good summaryRagingDragon2009/02/03 11:46 PM
                                                                                                      Terminology seems too fuzzy to meslacker2009/02/04 05:06 PM
                                                                                                        Terminology seems too fuzzy to meMichael S2009/02/05 01:05 AM
                                                                                                          Terminology seems too fuzzy to meUngo2009/02/05 01:15 PM
                                                                                                          Terminology seems too fuzzy to meslacker2009/02/05 02:19 PM
                                                                                                            Terminology seems too fuzzy to meHoward Chu2009/02/05 04:44 PM
                                                                                          Like Duhiz2009/01/30 02:03 AM
                                                                                          EAs (security labels) hosed me badlyanon2009/01/30 09:48 PM
                                                                                        Extended Attributes in ActionRagingDragon2009/01/29 09:31 PM
                                                                                      Extended Attributes in Actionanonymous2009/01/29 08:13 PM
                                                                                      Extended Attributes in ActionHoward Chu2009/01/29 09:38 PM
                                                                                        Extended Attributes in Actionslacker2009/01/30 11:24 AM
                                                                                          Extended Attributes in Actionanon2009/01/30 05:50 PM
                                                                                      Extended Attributes in ActionEtienne Lehnart2009/01/30 12:22 AM
                                                                                      Extended Attributes in ActionRob Thorpe2009/01/30 12:39 PM
                                                                                        Extended Attributes in Actionslacker2009/01/30 01:16 PM
                                                                                          Extended Attributes in Actionanon2009/01/30 06:03 PM
                                                                                          Extended Attributes in ActionHoward Chu2009/01/30 11:22 PM
                                                                                            Extended Attributes in Actionrwessel2009/01/31 12:08 AM
                                                                                              Extended Attributes in Actionanonymous2009/01/31 12:22 AM
                                                                                                Extended Attributes in Actionrwessel2009/01/31 12:56 AM
                                                                                                  ScalingDean Kent2009/01/31 09:04 AM
                                                                                                    ScalingRob Thorpe2009/02/02 02:39 AM
                                                                                                      Scalingrwessel2009/02/02 11:41 AM
                                                                                                        ScalingHoward Chu2009/02/02 12:30 PM
                                                                                                          ScalingDean Kent2009/02/02 02:27 PM
                                                                                                            ScalingRob Thorpe2009/02/03 05:08 AM
                                                                                                              ScalingDean Kent2009/02/03 07:38 AM
                                                                                                              Scalingrwessel2009/02/03 02:34 PM
                                                                                                                ScalingRagingDragon2009/02/03 10:46 PM
                                                                                                    in defense of software that does not scaleMatt Sayler2009/02/03 11:27 AM
                                                                                                      in defense of software that does not scaleHoward Chu2009/02/03 12:03 PM
                                                                                                        in defense of software that does not scaleMatt Sayler2009/02/03 12:17 PM
                                                                                                        in defense of software that does not scaleRagingDragon2009/02/03 11:00 PM
                                                                                                          in defense of software that does not scaleMichael S2009/02/04 06:46 AM
                                                                                                            in defense of software that does not scaleRagingDragon2009/02/04 09:33 PM
                                                                                                      in defense of software that does not scaleDean Kent2009/02/03 12:17 PM
                                                                                                        in defense of software that does not scaleMatt Sayler2009/02/03 12:24 PM
                                                                                                        in defense of software that does not scaleVincent Diepeveen2009/02/04 10:43 AM
                                                                                                      in defense of software that does not scalerwessel2009/02/03 02:44 PM
                                                                                                        in defense of software that does not scaleanon2009/02/04 02:35 AM
                                                                                                          in defense of software that does not scaleCarlie Coats2009/02/04 05:24 AM
                                                                                                      Scaling with time vs. scaling from the beginning.mpx2009/02/05 01:57 AM
                                                                                                  Extended Attributes in ActionMichael S2009/01/31 10:33 AM
                                                                                                  Extended Attributes in Actionanon2009/01/31 10:37 PM
                                                                                          Extended Attributes in ActionJasonB2009/01/31 08:11 AM
                                                                                            Extended Attributes in ActionHoward Chu2009/01/31 11:43 AM
                                                                                              Extended Attributes in ActionJasonB2009/01/31 04:37 PM
                                                                                                Extended Attributes in ActionHoward Chu2009/02/02 02:42 PM
                                                                                                  Extended Attributes in ActionHoward Chu2009/02/02 02:44 PM
                                                                    The File AbstractionRob Thorpe2009/01/27 11:20 AM
                                                              The File AbstractionHoward Chu2009/01/27 12:28 AM
                                                                The File AbstractionMichael S2009/01/27 03:00 AM
                                                                The File AbstractionDean Kent2009/01/27 08:30 AM
                                                        The File AbstractionAndi Kleen2009/01/27 02:05 AM
                                  SLC vs. MLCMichel2009/01/12 06:54 PM
                                    SLC vs. MLCLinus Torvalds2009/01/12 07:38 PM
                                      SLC vs. MLCrwessel2009/01/13 12:52 AM
                                      SLC vs. MLCUngo2009/01/13 03:04 PM
                                        SLC vs. MLCWes Felter2009/01/13 05:42 PM
                                      SLC vs. MLCTruePath2009/01/25 05:05 AM
                    SLC vs. MLCUngo2008/08/21 12:54 PM
                    SLC vs. MLCAaron Spink2008/08/21 01:20 PM
                  MLC vs. SLCMichael S2008/08/21 08:57 AM
                First Dunnington benchmark resultsrwessel2008/08/21 10:40 AM
              First Dunnington benchmark resultsAaron Spink2008/08/21 03:18 AM
      First Dunnington benchmark resultsEtienne Lehnart2008/08/20 04:38 AM
  Will x86 dominate big iron?Tom W2008/08/19 10:10 PM
    Will x86 dominate big iron?Jesper Frimann2008/08/20 12:28 AM
      Will x86 dominate big iron?Tom W2008/08/20 03:42 PM
        Will x86 dominate big iron?David Kanter2008/08/21 01:13 AM
          Will x86 dominate big iron?Joe Chang2008/08/21 06:54 AM
            Will x86 dominate big iron?asdf2008/08/22 01:18 PM
              Will x86 dominate big iron?Dean Kent2008/08/22 07:54 PM
        Will x86 dominate big iron?Jesper Frimann2008/08/22 09:48 AM
          Will x86 dominate big iron?Tom W2008/08/24 01:06 AM
            Will x86 dominate big iron?Michael S2008/08/24 04:19 AM
              Will x86 dominate big iron?Dean Kent2008/08/24 09:30 AM
                Will x86 dominate big iron?Paul2008/08/24 11:16 AM
                  Will x86 dominate big iron?Dean Kent2008/08/24 12:37 PM
                Will x86 dominate big iron?Michael S2008/08/25 12:53 AM
        Will x86 dominate big iron?someone2008/08/22 10:19 AM
          Will x86 dominate big iron?aaron spink2008/08/23 02:56 AM
            Will x86 dominate big iron?Michael S2008/08/23 09:58 AM
              Will x86 dominate big iron?someone2008/08/23 01:51 PM
            Will x86 dominate big iron?someone2008/08/23 01:55 PM
              Will x86 dominate big iron?Aaron Spink2008/08/23 04:52 PM
                Will x86 dominate big iron?anonymous2008/08/23 05:28 PM
                  Will x86 dominate big iron?Dean Kent2008/08/23 06:12 PM
                    Off road and topicEduardoS2008/08/23 06:28 PM
                  Will x86 dominate big iron?someone2008/08/23 06:26 PM
                    Will x86 dominate big iron?Dean Kent2008/08/23 09:40 PM
                    Will x86 dominate big iron?anonymous2008/08/24 01:46 AM
                  Off road and topicDavid W. Hess2008/08/24 03:24 AM
                    Off road and topicAaron Spink2008/08/24 04:14 AM
  Beckton vs. DunningtonMr. Camel2008/08/22 06:30 AM
    Beckton vs. Dunningtonjokerman2008/08/22 12:12 PM
    Beckton vs. DunningtonMr. Camel2009/05/29 10:16 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell green?