By: David Kanter (dkanter.delete@this.realworldtech.com), April 16, 2011 8:42 am
Room: Moderated Discussions
Kevin G (kevin@cubitdesigns.com) on 4/16/11 wrote:
---------------------------
>MS (ms@lostcircuits.com) on 4/15/11 wrote:
>---------------------------
>>
>>I personally prefer [H] for graphics card performance, >>Brent is very good at what he does.
>
>I enjoy HardOCP's content for graphics cards but their testing methodology is the
>exact opposite of what you'd base this type of analysis >on. Their real world based
>game play analysis introduces a variance that could skew >results.
Yes, this is a huge issue. In looking at benchmarks, the first thing I'd really need to do is eliminate all sources of extraneous and unwanted variability. Note that from a statistical standpoint, you do want variability, but in the dimensions that matter (and can accommodate it).
If I'm not running the same workload against each card, then I have no business making a comparison.
>Stuff like 3DMark
>can produce reliably repeatable results. Secondly the >bulk of their analysis stems
>from the maximum image quality they can attain in a test >while maintaining playable
>frame rates. From a gamer's perspective this works out rather well trying to pick
>between two cards, especially when frame rates are often similar at the same settings.
>They do offer some apples-to-apples comparison but that >isn't what they base their
>conclusions on. Thirdly, I don't believe that the data >offered by HardOCP isn't
>extensive enough to do the kind of analysis David here is >attempting.
There's actually a more subtle criteria than just extensive. It needs to be extensive and in usable form.
Anand does a lot of testing, but I'm unsure of his methods and the data is in a useless format.
The notebookcheck guys have data that is in a very tractable form! I read a bunch of their reviews, and they all seem to have a reasonable approach.
To put it in perspective, if I had to work from [H] or AT, the article would have never been written. I simply have better things to do with my time than try and manually reformat data from a website into a spreadsheet.
My preference would be to work from Scott's data at TR, but he doesn't use 3DMark.
David
---------------------------
>MS (ms@lostcircuits.com) on 4/15/11 wrote:
>---------------------------
>>
>>I personally prefer [H] for graphics card performance, >>Brent is very good at what he does.
>
>I enjoy HardOCP's content for graphics cards but their testing methodology is the
>exact opposite of what you'd base this type of analysis >on. Their real world based
>game play analysis introduces a variance that could skew >results.
Yes, this is a huge issue. In looking at benchmarks, the first thing I'd really need to do is eliminate all sources of extraneous and unwanted variability. Note that from a statistical standpoint, you do want variability, but in the dimensions that matter (and can accommodate it).
If I'm not running the same workload against each card, then I have no business making a comparison.
>Stuff like 3DMark
>can produce reliably repeatable results. Secondly the >bulk of their analysis stems
>from the maximum image quality they can attain in a test >while maintaining playable
>frame rates. From a gamer's perspective this works out rather well trying to pick
>between two cards, especially when frame rates are often similar at the same settings.
>They do offer some apples-to-apples comparison but that >isn't what they base their
>conclusions on. Thirdly, I don't believe that the data >offered by HardOCP isn't
>extensive enough to do the kind of analysis David here is >attempting.
There's actually a more subtle criteria than just extensive. It needs to be extensive and in usable form.
Anand does a lot of testing, but I'm unsure of his methods and the data is in a useless format.
The notebookcheck guys have data that is in a very tractable form! I read a bunch of their reviews, and they all seem to have a reasonable approach.
To put it in perspective, if I had to work from [H] or AT, the article would have never been written. I simply have better things to do with my time than try and manually reformat data from a website into a spreadsheet.
My preference would be to work from Scott's data at TR, but he doesn't use 3DMark.
David
Topic | Posted By | Date |
---|---|---|
New Article: Predicting GPU Performance for AMD and Nvidia | David Kanter | 2011/04/11 11:55 PM |
Graph is not red-green colorblind friendly (NT) | RatherNotSay | 2011/04/12 03:51 AM |
Fixed | David Kanter | 2011/04/12 08:46 AM |
New Article: Predicting GPU Performance for AMD and Nvidia | James | 2011/04/12 12:30 PM |
New Article: Predicting GPU Performance for AMD and Nvidia | David Kanter | 2011/04/12 02:51 PM |
Try HD6450 or HD6850 | EduardoS | 2011/04/12 03:31 PM |
Try HD6450 or HD6850 | David Kanter | 2011/04/13 10:25 AM |
Try HD6450 or HD6850 | EduardoS | 2011/04/13 03:20 PM |
of cause | Moritz | 2011/04/14 08:03 AM |
of cause | EduardoS | 2011/04/14 01:55 PM |
Barts = 5D | Moritz | 2011/04/14 09:26 PM |
Barts = 5D | Antti-Ville Tuunainen | 2011/04/15 12:38 AM |
Limiting fixed function units | Moritz | 2011/04/15 04:28 AM |
Limiting fixed function units | Vincent Diepeveen | 2011/04/20 02:38 AM |
lack of detail | Moritz | 2011/04/20 09:24 AM |
lack of detail | EduardoS | 2011/04/20 11:45 AM |
gpgpu | Vincent Diepeveen | 2011/04/16 02:10 AM |
gpgpu | EduardoS | 2011/04/17 12:31 PM |
gpgpu | Groo | 2011/04/17 12:58 PM |
gpgpu | EduardoS | 2011/04/17 01:08 PM |
gpgpu | Ian Ameline | 2011/04/18 03:55 PM |
gpgpu | Ping-Che Chen | 2011/04/19 12:59 AM |
GPU numerical compliance | Sylvain Collange | 2011/04/19 11:38 AM |
GPU numerical compliance | Vincent Diepeveen | 2011/04/20 02:17 AM |
gpgpu | Vincent Diepeveen | 2011/04/20 02:02 AM |
gpgpu and core counts | Heikki Kultala | 2011/04/20 04:41 AM |
gpgpu and core counts | Vincent Diepeveen | 2011/04/20 05:52 AM |
gpgpu and core counts | none | 2011/04/20 07:05 AM |
gpgpu and core counts | EduardoS | 2011/04/20 11:36 AM |
gpgpu and core counts | Heikki Kultala | 2011/04/20 10:16 AM |
gpgpu and core counts | EduardoS | 2011/04/20 11:34 AM |
gpgpu and core counts | Heikki Kultala | 2011/04/20 07:24 PM |
gpgpu and core counts | EduardoS | 2011/04/20 08:55 PM |
gpgpu and core counts | Heikki Kultala | 2011/04/21 06:48 AM |
gpgpu and core counts | EduardoS | 2011/04/22 01:41 PM |
AMD Compute and Texture Fetch | David Kanter | 2011/04/21 10:42 AM |
AMD Compute and Texture Fetch | Vincent Diepeveen | 2011/04/22 01:14 AM |
AMD Compute and Texture Fetch | David Kanter | 2011/04/22 10:53 AM |
AMD Compute and Texture Fetch | EduardoS | 2011/04/22 01:46 PM |
AMD Compute and Texture Fetch | David Kanter | 2011/04/22 02:02 PM |
AMD Compute and Texture Fetch | EduardoS | 2011/04/22 02:18 PM |
AMD Compute and Texture Fetch | anon | 2011/04/22 03:30 PM |
AMD Compute and Texture Fetch | David Kanter | 2011/04/22 09:17 PM |
gpgpu and core counts | Vincent Diepeveen | 2011/04/20 12:12 PM |
gpgpu and core counts | Heikki Kultala | 2011/04/21 10:23 AM |
gpgpu and core counts | Vincent Diepeveen | 2011/04/22 02:11 AM |
Keep the crazy politics out of this | David Kanter | 2011/04/22 08:39 AM |
Keep the crazy politics out of this | Vincent Diepeveen | 2011/04/22 09:12 AM |
Keep the crazy politics out of this | David Kanter | 2011/04/22 10:44 AM |
gpgpu and core counts | Jouni Osmala | 2011/04/22 11:06 AM |
gpgpu | EduardoS | 2011/04/20 11:59 AM |
gpgpu | Vincent Diepeveen | 2011/04/20 12:37 PM |
gpgpu | EduardoS | 2011/04/20 05:27 PM |
gpgpu | Vincent Diepeveen | 2011/04/21 02:06 AM |
gpgpu | EduardoS | 2011/04/22 02:00 PM |
New Article: Predicting GPU Performance for AMD and Nvidia | PiedPiper | 2011/04/12 10:05 PM |
New Article: Predicting GPU Performance for AMD and Nvidia | David Kanter | 2011/04/12 10:42 PM |
New Article: Predicting GPU Performance for AMD and Nvidia | MS | 2011/04/15 05:04 AM |
New Article: Predicting GPU Performance for AMD and Nvidia | Kevin G | 2011/04/16 02:25 AM |
New Article: Predicting GPU Performance for AMD and Nvidia | David Kanter | 2011/04/16 08:42 AM |
New Article: Predicting GPU Performance for AMD and Nvidia | Vincent Diepeveen | 2011/04/20 02:20 AM |
memory | Moritz | 2011/04/14 09:03 PM |
memory - more | Moritz | 2011/04/15 11:11 PM |
New Article: Predicting GPU Performance for AMD and Nvidia | Kevin G | 2011/04/14 11:30 AM |