Linus Torvalds (torvalds.delete@this.linux-foundation.org) on July 13, 2020 11:06 am wrote:
> If you have to normalize, do it only at literal "untrusted user entry". In that graphical
> UI box that the user actually types in. Or that network input side processing that
> expects a character string. Do the normalization there, and do it before you start
> parsing anything. If you do it any later, you're screwed and did it wrong.
>

Good advice, but also wrong.

Almost every boundary between two pieces of software is "untrusted user entry", either two things communicating across a network, or something calling into library. Our long history of input security vulnerabilities comes from failing to treat data at such boundaries as "untrusted".

A good example is a recent Exim vulnerability. It checked "hostnames" when they came from DNS and config files, but trusted that hostnames it got from the OpenSSL library were validated. But they weren't, so a corrupted SNI field could hack the email server. This vuln worked with not only OpenSSL, but also GnuTLS (as Exim support many different SSL libraries).

The underlying problem was that it's not clear what the SNI field is supposed to contain. Should SSL libraries treat it as a blob and simply pass it on through? Or should they validate that it conforms to hostname rules? Should Unicode names be allowed? Or should only the DNS punycode names be allowed? It's not clearly specified, so the SSL libraries treated it as a blob and passed it on through. And Exim didn't validate it either, so bug.

Thus, everybody that writes libraries or uses libraries needs to be concerned with validating UTF8 strings. Which is everybody.

How and what to do is a dark art rather than a simple science. The above posts talk about "normalization" which is almost always a bad idea these days. It should be "validated", such that if it isn't already in a normalized form, it should be rejected.

When you accept bad input, you get locked into legacy of being forced to accept that bad input until the end of time, because when you issue a software patch that makes validation stricter, customers complain. Going the other way, loosening restrictions to accept more input, is much easier.

Accepting bad input also means corrupting that input. You pat yourself on the back because 99% of the time your guess right at what the customer wanted, but you are ignoring the other 1% of the time when you guessed wrong and Very Bad Things happened. It took us 20 years to clean up the HTML standards to get rid of this flawed thinking.

The point being is that every programmer should be afraid of messing with UTF8 encoded data, that much of the time, they should treat it as a blob. But at the same time, this simple advice fails to recognize the practical reality that we have to mess with. Programmers should avoid changing it wherever possible, but they often have to reject bad input.