Travis (travis.downs.delete@this.gmail.com) on August 3, 2018 7:58 pm wrote:
> I think vectorization of C and C++ isn't a total dead end, and we'll see more and more pushed by compiler
> vendors - but it definitely hasn't panned out as some magic cure to the end of Moore's law.

C++ is a total dead end.
I would recommend a real language like Swift or Go instead, but autovectorization is not available yet.

> I do think some of the projects are related to the low semantic level that C and C++
> operate at, which limits some opportunities - but that's the only problem. I see that
> the envelope for success for auto-vectorization has limits in every direction:
>
> 1) People who really care about performance of a few specific kernels, like HPC, media encoding,
> BLAS libraries, whatever, aren't going to use auto-vectorization in the first place in it is (a)
> too fragile and (b) they already know exactly how they want their code to vectorize.

#pragma clang loop vectorize(enable)
#pragma clang loop vectorize_width(2) interleave_count(2)

And a long list of variants for loops so that you do not have to write assembly.
https://llvm.org/docs/Vectorizers.html

> They are just
> using hand-written assembly or (less likely) intrinsics to get exactly what they want.
>
> This code is a very small % of overall code "by volume" but it is a significant amount of code
> weighted by runtime, and so this highly SIMD-friendly code will not be auto-vectorized.
>
> 2) The auto-vectorization abilities of various compilers varies widely and not in totally ordered
> way. That is, it's not like you can say compiles have some relationship like X also by Z. If that were the case at least you just need to target your stuff to X and it will
> auto-vectorized everywhere. Because it doesn't work like that, if you target multiple compilers
> you have this giant clusterfuck of always checking every compiler after every change.

LLVM is so much better than the alternatives I do not see why you would use any other compiler.

> 3) If you really rely on auto-vectorization you simply have to check that it worked, otherwise you're
> just opening yourself up to big performance regressions. If you are smart enough to read the assembly
> and see that it worked however, maybe you should just write it in the first place (or if you are lazy,
> copy paste the compiler's autovectorized code). In some way the "do it yourself" approach scales better:
> you only have do to it once, and perhaps update it for new ISAs, but with the check approach you have
> to check any time you change any surrounding code or compiler versions, etc.
>
> 4) Although often leading to worse results than assembly, intrinsics really lower the bar for
> writing it yourself. If the choice was between auto-vectorization and adding assembly to your
> build (and dealing with calling convention differences, debug info, eh_frame/SEH and other complications
> you get in assembly) - you would probably see a lot more willingness to try to bend the auto-vectorizer
> to your will. Intrinsics are much easier though: they avoid almost all of the problems of assembly
> and you can add them a few at a time. They are a gateway drug.
>
> 5) Even if the compilers had really good auto-vectorizers (they don't), they lack the global information needed
> to actually make hard decisions about what and how to autovectorize. The whole "AVX turbo" stuff really throws
> a wrench into optimization decisions: you better be damn sure you know what you are doing before you introduce
> some AVX or AVX512 instructions and slow the rest of the application by 30% or whatever. This kind of "global"
> problem is kind of unprecedented. Well you always had some kind of global tradeoffs in terms of code size/speed
> tradeoffs, but the impact wasn't anything like this: deciding to vectorize one loop that doesn't actually
> have many iterations might slow down the CPU for millions of instructions.
>
> Intel, who out of everyone has the most to win by actually having heavy use of all their new ISA,
> had to introduce a change to their compiler which basically disables most AVX512 vectorization
> unless you use a special compiler switch because vectorization was slowing everything down.

C++ is too pathetic to ever auto-vectorize, you have to use #pragma’s to tell the compiler what to do. Period.

> I don't know if this will ever be solved in a generic way (no PGO doesn't really do
> it). An "easy" by-hand solution is just to annotate the methods you want to vectorize,
> but that kind of chips away at the dream of autovectorization everywhere.

Eventually a real language will add autovectorization and people will dump C++.

> 6) SIMD kind of lives in this weird netherworld where it is rarely "best" for anything: if your code is
> really SIMD-friendly, it will often do even better on a GPU or GPU-like thing. If it's some ML thing it
> will do better on an ML-specific chip, etc. Even if it's more SIMD-friendly than GPU-friendly, the huge
> gap in raw power often results it being faster on GPU even if you achieve a lower percentage of peak or
> whatever. So SIMD mostly gets used because the programming model is way simpler: every CPU has it, you
> don't need to fight with drivers or GPU differences, there is an obvious time-sharing model already for
> the CPU. Mostly everything just works. GPU is a lot of work and also a different skill set... but GPU is
> always there lurking, ready to slice off a big part of the envelope where SIMD might be ideal.
>
> This is actually more a restriction of the SIMD emvelope, not really auto-vectorization
> - but anything that restricts SIMD use-cases also restricts auto-vectorization wins.
>
> 7) A lot of the really vectorizable stuff in an average application are things like memcpy, string operations
> and so on that are already vectorized in by hand in libc. So profiling various random applications do you
> do see a fair amount of AVX2 from time to time: but it's coming from libc code. Even C++ is designed in such
> a way that many higher level copy type functions can use memcpy as the underlying primitive if the objects
> are compatible. Compilers even try to recognize if have written a memcpy loop yourself and replace it by memcpy.
> If this wasn't the case, these kinds of things would be obvious big wins for vectorization.
>
> So I think auto-vectorization has room to succeed: perhaps more so in higher-level level languages
> that solve many inherent problems that C and C++ have (such as unknown aliasing, unknown alignment,
> low level operations) and especially it JITed languages which have the huge advantage of being
> solving the "is it worth it to take the AVX-turbo hit?" dynamically at runtime.
>
> The other fact is that auto-vectorizers still really just suck. I don't want to bash the compiler writers too
> much: it's a hard, mostly thankless, problem and they have come a long way, but compilers still generally really,
> really shitty autovectorized code on a regular basis. They can turn a 5-line function into 1,000 instructions.
> They can use hundreds of instructions on prologue and epilogue code, but then not unroll the main loop at all,
> losing a possible 300% speedup by saving a few instructions. They can just do some insane shuffling stuff: shuffling
> the same data 4 or 5 times just to do a single vectorized op. They might build up an entire vector one-element-at-a-time
> with slow pinsr instructions and then only do a few ops, and on and on.
>
> I think the biggest benefit I've seen from automatic use of vector instructions, that actually almost
> always works well, isn't the loop vectorizer at all but the use of 16 or 32 byte mov instructions
> for fixed sized copies, e.g., of structures. These show up all over the place and you get a nice
> win there (and at least these are in the "cheap tier" when it comes to AVX-turbo slowdowns).
>