Michael S (already5chosen.delete@this.yahoo.com) on July 15, 2020 11:05 am wrote:
>
> According to my understanding, it's not about Linux using either NFC or NFD.

Correct. Any normalization is fundamentally wrong in a filesystem.

The same way you don't want to change case, you really don't want to normalize. EVER.

It's really exactly the same issue.

Now, there are multiple reasons not to normalize, but the fundamental one is that you're losing information that can be relevant.

To pick one example: in Swedish, the letter 'ä' really is one letter, it has absolutely nothing to do with "a" combined with the diaeresis character. It's not an 'a' that is pronounced differently or has some other thing associated with it. It's literally its own character.

In contrast, in some other situations and other languages, the markings on top of characters (or bottom of) really are "modifiers" to the character, and people may intentionally use the decomposed form (although they rarely actually do, afaik).

So normalizing data that you don't understand how it is used (and by definition, an OS cannot know what the meaning of a filename is - that's a user choice), is truly and fundamentally broken.

And if some person tells you that they are the same, and by definition "canonically equivalent" and thus the differences don't matter, then that person doesn't understand Unicode. Being canonically equivalent does not mean that they are the same, and have to be treated the same. It means that they look the same when printed out. Nothing more, nothing less.

So canonical equivalence doesn't mean "equality", and it doesn't mean that normalizing doesn't matter. But some people get confused about it, because they think that "looks the same" means "is the same". Any technology person should know that isn't true. There are deep and fundamental differences, the same way ASCII 'A' looks and acts exactly like EBCDIC 'A', but has a fundamental encoding difference despite that.

You can't convert text from ASCII to EBCDIC and say that the end result is the same. It's not. They simple are not the same - not even if they are 100% equivalent, just different encodings of the exact same text, and should be printed out exactly the same.

And exactly as with ASCII and EBCDIC, the Unicode normal form conversions aren't reversible.

Is normalizing always wrong? No. There are places and times when you want to do it, but they are very very very rare. And you have to truly and fundamentally understand the context to do it.

So most commonly, you do normalization inside things that do comparisons between two strings, and the comparison is defined to be a normalizing one (even unicode comparisons do not have to be for equivalence! Doing "strcmp()" on UTF-8 isn't wrong). Or you may want to normalize to a combined character as part of picking the actual glyph: when you do ligatures between the letters 'f' and 'i', you should not be doing it as part of the context if the next character is a diearesis that combines with the 'i'.

(Ok, I'm not sure the latter case really counts as a normalization: it may actually be done that way, but I suspect in the big picture that's just all about the font layout rather than normalization per se).

But doing normalization at an interface where you do not understand the context is absolutely always wrong.

(Note that this is different from "malformed UTF-8", which you should also never normalize, you should either just reject that outright, or not consider to be Unicode at all. The most common reason for "malformed UTF-8" is that it simply isn't UTF-8 at all, and you have content that is some other format entirely, and you need to do a locale-aware conversion to Unicode, and turn it into UTF-8 that way).

Anyway, in almost all situations, the thing people want is NFC ("composed", "compact" or "cotrrect" - you choose the meaning of the "C"). It's the most common one, and it's the one you should use if you don't know what you're doing, and have no particular preferences, and just want to say "my data is in this form".

There are people out there (I think they are all Apple apologists) that claim that NFD is preferred because it's "faster". That's complete and utter BS. If you hear that claim, you can dismiss that person as not knowing what he's doing. That person probably had one particular problem where the decomposed form was simpler, and then decided NFD is the way to go.

But if NFD is the better one for your particular case, and you understand the conctext, by all means use NFD. NFD isn't wrong per se. It's only wrong when used blindly and when you then expose others to it. The most common reason for NFD may be to internally use it when you compare two strings, and you know that equivalence is what you want to use as the comparison criteria.

But even when you use NFD, and even when you know that your comparison criteria is "Unicode equivalence" (probably combined with case insensitivity), you don't change your source data that somebody else is using to do that! You just use it inside your own particular case where it made sense.

In a filesystem, NFD most certainly is not the way to go, for several reasons:

- don't normalize. Christ, people!

- it's less compact

- of the two main normalization choices, it's simply the wrong one

- you don't generally have to normalize even for case-insensitive comparisons

In particular, I think that what OS X (probably) does in its file handling code is to normalize the input to NFD at entry, because that way you normalize once and are done. It's slightly easier, but it means you do extra allocations and copies, and it's stupid and wrong.

And once you've done that stupid and wrong normalization, now you don't even have the original filename any more, so you save that wrong and normalized copy on disk.

Stupid.

Ignorant.

Wrong.

The smart thing to do is to avoid normalizing as long as possible, because even when you do file lookups, you can do the exact (non-normalized) match first, and that will wotk 99% of the time. And you did no extra work, so your 99% is faster.

When people have a file called "Banana" on OS X, it's true that they can look it up using the name "bAnAnA" too. But let's be honest: they don't. They open the file "Banana".

And then, because you have to get the "bAnAnA" case right too for the insane user, if you didn't have an exact match, you now go to the "slow compare" thing, and that one is likely better also done without ever normalizing either of the strings, but instead doing the comparison one normalized character at a time. Again, it's actually generally faster and cheaper to do it that way, partly because most of the time, the normalization is really simple (ie a lot of filenames are in fact plain ASCII, even in non-ASCII places - think all the system directories etc).

To get us back to vectorization, it's actually trivially doable to do all the normal US ASCII cases with a vectorized one (turning all ASCII letters from lower case to upper case is not hard, neither is it hard to test "are all these bytes US ASCII"). So the "handle all the really cheap and common cases without ever normalizing anything at all" is literally the fastest thing you can do in almost all cases, and it's often the simplest thing too, because all your fast-paths never need to care or know, or worry about extra allocations because the strings change length.

Btw, Linux doesn't normalize, and Linux filesystem accesses are generally much faster than OS X. And simpler to work with.

And look - Linux didn't corrupt any user data in the process!

Normalizing at a filesystem level really is wrong. Quod erat demonstrandum.

Linus