I think vectorization of C and C++ isn't a total dead end, and we'll see more and more pushed by compiler vendors - but it definitely hasn't panned out as some magic cure to the end of Moore's law.

I do think some of the projects are related to the low semantic level that C and C++ operate at, which limits some opportunities - but that's the only problem. I see that the envelope for success for auto-vectorization has limits in every direction:

1) People who really care about performance of a few specific kernels, like HPC, media encoding, BLAS libraries, whatever, aren't going to use auto-vectorization in the first place in it is (a) too fragile and (b) they already know exactly how they want their code to vectorize. They are just using hand-written assembly or (less likely) intrinsics to get exactly what they want.

This code is a very small % of overall code "by volume" but it is a significant amount of code weighted by runtime, and so this highly SIMD-friendly code will not be auto-vectorized.

2) The auto-vectorization abilities of various compilers varies widely and not in totally ordered way. That is, it's not like you can say compiles have some relationship like X < Y < Z meaning that everything auto-vectorized by X is always auto-vectorized by Y and Z, and everything by Y also by Z. If that were the case at least you just need to target your stuff to X and it will auto-vectorized everywhere. Because it doesn't work like that, if you target multiple compilers you have this giant clusterfuck of always checking every compiler after every change.

3) If you really rely on auto-vectorization you simply have to check that it worked, otherwise you're just opening yourself up to big performance regressions. If you are smart enough to read the assembly and see that it worked however, maybe you should just write it in the first place (or if you are lazy, copy paste the compiler's autovectorized code). In some way the "do it yourself" approach scales better: you only have do to it once, and perhaps update it for new ISAs, but with the check approach you have to check any time you change any surrounding code or compiler versions, etc.

4) Although often leading to worse results than assembly, intrinsics really lower the bar for writing it yourself. If the choice was between auto-vectorization and adding assembly to your build (and dealing with calling convention differences, debug info, eh_frame/SEH and other complications you get in assembly) - you would probably see a lot more willingness to try to bend the auto-vectorizer to your will. Intrinsics are much easier though: they avoid almost all of the problems of assembly and you can add them a few at a time. They are a gateway drug.

5) Even if the compilers had really good auto-vectorizers (they don't), they lack the global information needed to actually make hard decisions about what and how to autovectorize. The whole "AVX turbo" stuff really throws a wrench into optimization decisions: you better be damn sure you know what you are doing before you introduce some AVX or AVX512 instructions and slow the rest of the application by 30% or whatever. This kind of "global" problem is kind of unprecedented. Well you always had some kind of global tradeoffs in terms of code size/speed tradeoffs, but the impact wasn't anything like this: deciding to vectorize one loop that doesn't actually have many iterations might slow down the CPU for millions of instructions.

Intel, who out of everyone has the most to win by actually having heavy use of all their new ISA, had to introduce a change to their compiler which basically disables most AVX512 vectorization unless you use a special compiler switch because vectorization was slowing everything down.

I don't know if this will ever be solved in a generic way (no PGO doesn't really do it). An "easy" by-hand solution is just to annotate the methods you want to vectorize, but that kind of chips away at the dream of autovectorization everywhere.

6) SIMD kind of lives in this weird netherworld where it is rarely "best" for anything: if your code is really SIMD-friendly, it will often do even better on a GPU or GPU-like thing. If it's some ML thing it will do better on an ML-specific chip, etc. Even if it's more SIMD-friendly than GPU-friendly, the huge gap in raw power often results it being faster on GPU even if you achieve a lower percentage of peak or whatever. So SIMD mostly gets used because the programming model is way simpler: every CPU has it, you don't need to fight with drivers or GPU differences, there is an obvious time-sharing model already for the CPU. Mostly everything just works. GPU is a lot of work and also a different skill set... but GPU is always there lurking, ready to slice off a big part of the envelope where SIMD might be ideal.

This is actually more a restriction of the SIMD emvelope, not really auto-vectorization - but anything that restricts SIMD use-cases also restricts auto-vectorization wins.

7) A lot of the really vectorizable stuff in an average application are things like memcpy, string operations and so on that are already vectorized in by hand in libc. So profiling various random applications do you do see a fair amount of AVX2 from time to time: but it's coming from libc code. Even C++ is designed in such a way that many higher level copy type functions can use memcpy as the underlying primitive if the objects are compatible. Compilers even try to recognize if have written a memcpy loop yourself and replace it by memcpy. If this wasn't the case, these kinds of things would be obvious big wins for vectorization.

So I think auto-vectorization has room to succeed: perhaps more so in higher-level level languages that solve many inherent problems that C and C++ have (such as unknown aliasing, unknown alignment, low level operations) and especially it JITed languages which have the huge advantage of being solving the "is it worth it to take the AVX-turbo hit?" dynamically at runtime.

The other fact is that auto-vectorizers still really just suck. I don't want to bash the compiler writers too much: it's a hard, mostly thankless, problem and they have come a long way, but compilers still generally really, really shitty autovectorized code on a regular basis. They can turn a 5-line function into 1,000 instructions. They can use hundreds of instructions on prologue and epilogue code, but then not unroll the main loop at all, losing a possible 300% speedup by saving a few instructions. They can just do some insane shuffling stuff: shuffling the same data 4 or 5 times just to do a single vectorized op. They might build up an entire vector one-element-at-a-time with slow pinsr instructions and then only do a few ops, and on and on.

I think the biggest benefit I've seen from automatic use of vector instructions, that actually almost always works well, isn't the loop vectorizer at all but the use of 16 or 32 byte mov instructions for fixed sized copies, e.g., of structures. These show up all over the place and you get a nice win there (and at least these are in the "cheap tier" when it comes to AVX-turbo slowdowns).