> > If everyone is 'cheating' by ignoring turbo time limits and/or using MCE, and consumers are likely to leave
> > those options on or turn them on themselves if not enabled by default, what should reviewers do?
> >
> There are two schools of thought here. One is that you are doing what the system is capable of doing even though
> it is cheating an only applicable to a specific benchmark or scenario. I don't subscribe to this one.

Stuff like MCE/ignoring time limits can be very broadly applicable though. Multithreaded applications often run for a while (say, video encoding).

> The other is you expose it, call it out, and trash the vendor/product in public for cheating. This is my preferred
> way. That said some are unwilling to do this because they know it will hurt their income stream.

My preferred way would be to test both. Show results with those 'enhancements' on, along with higher power consumption. But also show stock results to be fair to Intel/AMD. Let consumers know what vendors are doing, so they can make informed decisions.

> > Should they test the config customers will most likely be using?
> > Or turn off those options to get a 'proper' stock result?
> >
> Personally I would run it both ways, point out the level of cheating, trash the vendor
> who cheats, and recommend the other product. If a product cheats on X and is caught, are
> they doing other things elsewhere that weren't? Is it a good/safe/reliable product?
>
> > > What it comes down to is look a little deeper than the scores themselves, and
> > > point it out when people do wrong. If you don't, they will keep doing it.
>
> >
> > Yeah, keep pointing it out. But most people by now know to at least wait for independent
> > reviews, and to take initial manufacturer benchmarks with a grain of salt.
> >
> Another problem of late is companies are trying to short-circuit the independent review process.
> Some don't sample at all before sales, other do a teaser and pre-orders before reviews, and others
> just ignore the who idea of reviews. It usually blows back on them, but usually not for the short
> term so they 'win'. Long term they lose but that isn't in the calculus for the marketers
>
> -Charlie

Yeah, whoever pre-orders based on that limited info is taking a big gamble. How hard is it to wait for reviewers to publish reviews with more comprehensive benchmarks?