Geoff Langdale (geoff.langdale.delete@this.gmail.com) on July 11, 2020 7:49 pm wrote:
> OK, so I'm biased too - I used to work for Intel (not in hardware design, so I certainly don't have a vested
> interest in the particulars of AVX-512). And while I'm not an mouth-frothing Intel loyalist, I did invest a lot
> of time and trouble with SIMD programming and I think I've had a good deal of success making workloads that aren't
> from "Floating Point Benchmark Central Casting" go a lot faster with SIMD - specifically regular expression scanning
> (https://github.com/intel/hyperscan) and JSON parsing (https://github.com/lemire/simdjson).

In my experience there is a lot of code that can go significantly faster using SIMD. A common pattern is to find a conditional somewhere that gets mispredicted enough to be painful. If you replace it with branch-free code you often end up executing more µops and pay about as much overhead as you regain from being branch-free. Hardly worth the time to even try. But if you can do the branch-free thing 16-wide or 32-wide, you win by a lot.

Not only can you speed up your existing code, you can now do new things that previously never made sense because they were too expensive to contemplate. But a 16x or 32x reduction in overhead makes quite a difference.

The main downside is that you now have to modify your algorithms and write code in a language that Intel completely messed up. _mm256_permute2x128_si256? _mm256_add_epi16 and _mm256_add_epi32 instead of using the right type and typing "a += b"? Finding any detail in the programming interface that people don't intuitively dislike is hard.

But once you grit your teeth and accept the ugliness (or work around it), explicit parallelism specified by the programmer is much more efficient than OOO. I would predict that people will slowly modify their code to take advantage of it and the benefit of vector units will increase as they do.