Kevin G (kevin@cubitdesigns.com) on 4/16/11 wrote:
---------------------------
>MS (ms@lostcircuits.com) on 4/15/11 wrote:
>---------------------------
>>
>>I personally prefer [H] for graphics card performance, >>Brent is very good at what he does.
>
>I enjoy HardOCP's content for graphics cards but their testing methodology is the
>exact opposite of what you'd base this type of analysis >on. Their real world based
>game play analysis introduces a variance that could skew >results.

Yes, this is a huge issue. In looking at benchmarks, the first thing I'd really need to do is eliminate all sources of extraneous and unwanted variability. Note that from a statistical standpoint, you do want variability, but in the dimensions that matter (and can accommodate it).

If I'm not running the same workload against each card, then I have no business making a comparison.

>Stuff like 3DMark
>can produce reliably repeatable results. Secondly the >bulk of their analysis stems
>from the maximum image quality they can attain in a test >while maintaining playable
>frame rates. From a gamer's perspective this works out rather well trying to pick
>between two cards, especially when frame rates are often similar at the same settings.
>They do offer some apples-to-apples comparison but that >isn't what they base their
>conclusions on. Thirdly, I don't believe that the data >offered by HardOCP isn't
>extensive enough to do the kind of analysis David here is >attempting.

There's actually a more subtle criteria than just extensive. It needs to be extensive and in usable form.

Anand does a lot of testing, but I'm unsure of his methods and the data is in a useless format.

The notebookcheck guys have data that is in a very tractable form! I read a bunch of their reviews, and they all seem to have a reasonable approach.

To put it in perspective, if I had to work from [H] or AT, the article would have never been written. I simply have better things to do with my time than try and manually reformat data from a website into a spreadsheet.

My preference would be to work from Scott's data at TR, but he doesn't use 3DMark.

David