12 ways to fool the masses (HPC)

By: Paul A. Clayton (paaronclayton.delete@this.gmail.com), May 16, 2020 2:41 pm
Room: Moderated Discussions
I recently read a short 1991 paper by David H. Bailey "Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers"(PDF). While HPC performance comparisons seem not to be as popular as they were, some of the methods of described seem familiar from more recent comparisons.

  1. Quote only 32-bit performance results, not 64-bit results.

  2. Present performance figures for an inner kernel, and then represent these figures as the
    performance of the entire application.

  3. Quietly employ assembly code and other low-level language constructs.

  4. Scale up the problem size with the number of processors, but omit any mention of this

  5. Quote performance results projected to a full system.

  6. Compare your results against scalar, unoptimized code on Crays.

  7. When direct run time comparisons are required, compare with an old code on an
    obsolete system.

  8. If MFLOPS rates must be quoted, base the operation count on the parallel
    implementation, not on the best sequential implementation.

  9. Quote performance in terms of processor utilization, parallel speedups or MFLOPS
    per dollar.

  10. Mutilate the algorithm used in the parallel implementation to match the architecture.

  11. Measure parallel run times on a dedicated system, but measure conventional run
    times in a busy environment.

  12. If all else fails, show pretty pictures and animated videos, and don't talk about

Method 6 is a little less useful now as no system has such a substantial reputation, but comparing against the best using crippling configurations is still a "valid" technique.

Some of these methods are tempting to researchers because of time and funding constraints with an optimistic perspective on realistic scaling and application to real-world uses ("We can improve the compiler to provide code equivalent to hand-optimized assembly"). The under-valuing of negative results also encourages fudging the presentation (and even the data). If one really believes that one's technique has potential (or even just need to justify to oneself the time and effort spent on the project), the temptation to spin results is significant especially when everyone else is doing it. Print space and presentation time also constrain how much counter argument can be presented, though online university technical reports — which reduce such size constraints — often only or primarily increase the ease of replication and not the presentation of issues with the results.

The paper is only four pages and is a little more fun than just the list, but I thought even just the list was entertaining enough that some here would enjoy it.
 Next Post in Thread >
TopicPosted ByDate
12 ways to fool the masses (HPC)Paul A. Clayton2020/05/16 02:41 PM
  12 ways to fool the masses (HPC)Wes Felter2020/05/16 06:15 PM
  times have changedChester2020/05/17 03:55 PM
    times have changedanon2020/05/18 08:21 AM
      times have changedanon22020/05/18 08:34 PM
Reply to this Topic
Body: No Text
How do you spell purple?