David Kanter (dkanter.delete@this.realworldtech.com) on November 23, 2020 7:16 am wrote:
> > Since most/all of the subtests are irrelevant and meaningless for consumers, yeah I consider it
> > miles better. There's a reason SPEC isn't popular among review sites. No one cares about it.
>
> SPEC is unpopular because it's more difficult to obtain than
> other benchmarks and it's harder to run and understand.

Yeah, cost factors into it. But it's also bad for other reasons. Some of those were discussed in another thread (unrealistic workloads/compile settings).
- performance varies with compiler and compiler flags as well as hardware
- many of the tests just don't correspond to any consumer workload. Taking specint2017:
-- gcc - compile gcc itself
-- x264 - encode movie (but only write out every 200th frame, and use 1 mbps bitrate, and no asm)
-- perlbench - lots of email operations in perl, including spam filtering
-- mcf - single-depot vehicle scheduling
-- omnetpp - ethernet network simulation
-- xalancbmk - transform XML into HTML
-- deepsjeng - chess engine
-- leela - Go engine
-- exchange2 - sudoku generator, to test Fortran 95 array handling perf
-- xz - xz compression

Of these, the GCC test is nice to see. x264 would have been nice if not for its flaws. The rest...well are those really popular client applications?

gz, zip/deflate, and 7z are far more popular than xz compression.
perlbench sounds more like a server-side than client-side workload
mfc/omnetpp/exchange2/xalancbmk are super obscure
deepsjeng doesn't come to mind as a popular chess engine, unlike stockfish, crafty, and fritz.
I have no clue about leela.

Anyway, my main point is that pretty much any reputable review site has put together a better benchmark suite than SPEC. For example, take gamers nexus:
- Chromium compile - good to have, just like spec gcc
- Blender - good to have, also in specfp
- 7z compression/decompression - 7z is far more popular than xz
- Adobe Photoshop - lots of people use this
- Adobe Premier - also very popular for video editing
- V-Ray - meh, everyone I know uses Blender instead because it's free
- Handbrake x264 to x265 transcode - this is spec x264 done properly
- Cinebench R15 for validation - good because it's a free benchmark with results from lots of platforms and thus a good point of comparison to make sure something isn't horribly wrong with your system
- 10 games. Like it or not, the gaming market is growing, and a lot of people put gaming perf first when getting a system.