hobold (hobold.delete@this.vectorizer.org) on October 10, 2020 12:33 pm wrote:
> David Kanter (dkanter.delete@this.realworldtech.com) on October 10, 2020 10:36 am wrote:
>
> > Cinebench is a total rubbish benchmark!
>
> It happens to correlate very well with one or two workloads I care about. So for me it's
> been a useful benchmark, giving me a good idea of the performance I can expect.
>
> But like any performance metric (and like any benchmark), Cinebench tries to do an
> impossible thing when it maps "computing speed" to one single real number. That loses
> all information about specific strengths and weaknesses of different machines.

>
> Projecting a vector of many real numbers down to one single real number is always
> an information loss.

Mathematically, I am not certain that is the case. I do not understand infinities and have very little exposure to information theory, but I know that for a fixed/known number of prime numbers one can compose those prime numbers (by multiplication) into a single integer without loss of information (i.e., entry count and all primes provide a compression function). There was a (mediocre) science fiction story that used n = (prime_A)^x_A * (prime_B)^x_B * ... (prime_Ω)^x_Ω to encode a vast amount of information into a single integer. Since individual measurements have limited precision and range, concatenation of digits would convert N M-digit values into one M*N-digit value.

Yes, that is not what you meant. You meant something more like one single number that can be trivially used to accurately estimate performance for a workload based on a benchmark result.

Even with a well-suited, well-designed benchmark suite, using benchmark results to estimate performance for a workload is not trivial even if it might be straightforward (i.e., an algorithm exists). A formula like performance = (k₁ * result1<sup>C₁) * (k₂ * result2C₂) ... is straightforward and might provide accurate estimates across a broad range of systems, but it is not trivial (and requires generating the constants for the workload whose performance is being estimated).

> (And one could argue that the "true" performance profile
> isn't fully captured by a vector of microbenchmarks in the first place.)

Microbenchmarks are fragile because they do not account for interaction among hardware components. Larger, more complex subbenchmarks can reduce the prominence of such weaknesses, though such would be based on common interactions. A change in branch predictor accuracy could (unexpectedly) change the impact of L2 cache latency. A change in on-chip network topology and cache policies might accidentally improve performance of a subbenchmark that ordinarily highly correlates with the workload of interest (e.g., the workload of interest is sensitive to communication latency and the latency happens to go down for the subbenchmark because the communication pattern matches the hardware).

> Of course there can be many other ways in which a benchmark is bad.

I think one of the most common (valid) complaints are glass jaws and workload specificity. Examples of glass jaws include: Matrix300's vulnerability to sufficient cache size or compiler blocking-based optimization, libquantum's friendliness to autoparallelization (and the run-time rules not isolating this factor), art's AoS/SoA weakness, and (JIT) compilers aggressively removing "dead" code (result never used, result is compile-time constant) or doing aggressive strength reductions (Dhrystone?). Workload specificity merely constrains the breadth of relevance (in application space and somewhat in time — the "good enough"/no-longer-primary-constraint effect, hardware acceleration for mature common computation, change in algorithms, methods, or goals, etc.).

(For profile-guided optimization, designing the training data set to be appropriately similar to the testing data set seems challenging.)

Even careful benchmark design can introduce unexpected glass jaws where the return-on-investment for a change is substantially greater than expected. Distinguishing between generally useful optimizations and benchmark specials is not easy.

(I thought of composing a long comment on benchmarks but as I starting working on such I realized an adequate writing would take a lot of thought, research, and editing. The project is still tempting, but I do not feel up to the challenge at the moment.)

> But from that point of
> view the statement quoted above does not contain enough information for further discussion.

I agree. I am disappointed that David Kanter made such an unhelpful post. (It was at least concise, which is rarely a virtue of my posts.☺)</sup>