Klimax (danklima.delete@this.gmail.com) on July 11, 2013 3:10 am wrote:
> Wilco (Wilco.Dijkstra.delete@this.ntlworld.com) on July 11, 2013 1:04 am wrote:
> > Klimax (danklima.delete@this.gmail.com) on July 10, 2013 11:30 pm wrote:


> > I agree any benchmark where a few compiler tricks make such a huge difference to the score is utterly
> > broken. I'm sure it would be possible to implement the same trick(s) in GCC given enough effort.
> > However even if AnTuTu wasn't already irrelevant as a CPU benchmark, breaking it certainly has made
> > it completely useless, so it seems unlikely it is worth the effort. What do you prefer, 5-10% better
> > performance across most applications or doubling the scores on a few benchmarks?
>
> No argument about breaking. On the other hand, impact on real applications is much harder to measure.
> (Closest is IIRC PCMark and then question of weighting of points has to be answered)
>
> In any case it doesn't mean ARM have shown they sped up real world use cases...

I don't understand why you think ARM needs to show anything when it is Intel who is cheating... GCC never tried to be the benchmark king - as someone else already mentioned GCC focuses improving real world code. Do you disagree with that?

> > > Iin the end, it's story about GCC vs. ICC, which is known for
> > > a long time to be very unequal and badly written benchmark.
> > >
> > > So the only incorrect thing was about ARM compiler and rest stands.
> >
> > No - the story is about Intel's underhanded tactics of optimizing a closed source benchmark. The
> > source code is not available, so even upgrading to a newer GCC or changing the options isn't possible.
> > So how did Intel manage to replace GCC altogether? It seems likely they paid off AnTuTu.
> >
> > Wilco
> >
>
> No. It shows that Intel's optimization is still superior to others. (Closest one is VC not GCC/CLANG)

No, that would only be true if GCC and other compilers were also allowed to optimize the source code for this benchmark (which they were not), and if the optimization tricks were general enough to be useful in lots of code (which it's not for the trick Exophase mentioned).

ICC's sole purpose is to get good benchmark scores for Intel CPUs, and there is no doubt it is good at that. ICC may have more optimizations, but they are very benchmark specific. As a result ICC's compiler tricks do not speed up real world code. In tests I did in the past ICC performed similar or worse than GCC and VC++ on real code despite having a huge lead in SPEC.

In any case whatever ICC does or how well it scores is irrelevant: If you want to benchmark native Android performance, then GCC is the only choice in town as that is currently the only native Android compiler (with LLVM becoming a possible 2nd choice). Secretly replacing GCC binaries with ICC ones in Android benchmarks is cheating.

> One part of issue is, that nobody demonstrated that this particular optimization breaks upon
> small almost irrelevant change to code like it was done to Sunspider benchmark.
>
> (With such observed loop in disassembly it shouldn't be hard
> to replicate in C/C++ and do mods and observe ASM/binary...)

Since you seem to have access to ICC, why don't you try it yourself?

Wilco