Implications for linux page cache

By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), March 3, 2021 1:54 pm
Room: Moderated Discussions
anon (anon.delete@this.gmail.com) on March 3, 2021 11:54 am wrote:
> Generate checksum on write & verify before copying to user space buffer on every read?

That's not doable anyway, since shared mappings are a thing. But it would be a pointless operation even outside of that issue.

Honestly, the solution is

(a) admitting that there is no such thing as "perfect"

(b) making hardware fundamentally more reliable (ie ECC)

(c) end-to-end strong checksumming of data, and replication of the stuff that really matters

I won't go into (a). There are too many people who look for "perfect" solutions, dismissing things that help, and I think those people are naive, bordering on insanity.

I've gone into (b) extensively here before. No, it's not going to fix everything, but it's going to help a lot of cases.

And (c) is very much not about things like "read()" and "write()". Those are not end-to-end operations, and they fundamentally cannot know whether the data they are copying is reliable or not, because they don't know what the rules for the data is.

Doing checksums at those points is entirely pointless: what you checksum may not be the actual data, because the corruption could have happened before (ie when doing a "write()" system call, maybe the data was already corrupt in user space - you're just generating a completely meaningless checksum).

Note that checksums of data written to disk is a different thing: you're basically adding a protocol checksum between memory and the disk contents, and trying to protect against everything that can go wrong in between the two. That's an entirely different thing from checksumming when copying from RAM to RAM (ie writing to the page cache).

So (c) needs higher-level checksums by the programs that actually deal with long-lived data that people care about, by the applications that actually have semantic understanding of that data. Obviously you'd like to have some error recovery (which might be in the form of ECC, but honestly, at a higher level you most likely want it to be at a much higher level of redundancy entirely).

But that (a) is important. Accept it. You will never have a perfect system. Not in security, and not in the "corruption cannot happen" sense. All you can do is do a lot of mitigation (and the primary mitigation should always be noticing corruption).

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
CPU & Memory bit flipsGanon2021/03/03 09:05 AM
  Also "Silent Data Corruption"Adrian2021/03/03 10:42 AM
    Thanks for the referenceGanon2021/03/03 11:47 AM
  Implications for linux page cacheanon2021/03/03 11:54 AM
    Implications for linux page cacheLinus Torvalds2021/03/03 01:54 PM
      memory errorsblaine2021/03/03 02:53 PM
        memory errorsanon22021/03/03 05:30 PM
          memory errorsdmcq2021/03/04 05:16 AM
            memory errorsEtienne Lorrain2021/03/04 06:26 AM
              memory errorsdmcq2021/03/04 06:40 AM
                memory errorsEtienne Lorrain2021/03/04 06:58 AM
                  memory errorsdmcq2021/03/04 07:12 AM
                  memory errorsCarson2021/03/05 02:31 AM
                    memory errorsEtienne Lorrain2021/03/05 06:23 AM
                      memory errorsrwessel2021/03/05 07:48 AM
                      memory errorsdmcq2021/03/05 12:01 PM
                        memory errorsrwessel2021/03/05 12:23 PM
                          memory errorsdmcq2021/03/05 12:51 PM
                      memory errorsBrendan2021/03/05 11:38 PM
                      memory errorsCarson2021/03/06 01:35 AM
                        memory errorsCarson2021/03/06 06:24 AM
                memory errorsDavid Hess2021/03/04 01:44 PM
                  memory errorsrwessel2021/03/04 05:14 PM
                  memory errorsLinus Torvalds2021/03/04 08:21 PM
                    memory errorsanon22021/03/04 09:46 PM
                      memory errorsCarson2021/03/05 02:43 AM
                        memory errorsanon22021/03/05 07:55 AM
                    memory errorsgallier22021/03/05 02:22 AM
                  memory errorsdmcq2021/03/05 12:59 PM
                    memory errorsDavid Hess2021/03/06 04:27 AM
                      memory errorsCarson2021/03/06 06:44 AM
                      memory errorsGabriele Svelto2021/03/06 10:11 AM
                        memory errorsDavid Hess2021/03/06 10:28 AM
                          memory errorsMichael S2021/03/06 02:45 PM
              memory errorsDoug S2021/03/04 10:48 AM
                memory errorsMichael S2021/03/04 11:36 AM
              memory errorsJörn Engel2021/03/04 03:32 PM
                memory errorsLinus Torvalds2021/03/04 08:47 PM
                  memory errorsEtienne Lorrain2021/03/05 01:09 AM
                  memory errorsMichael S2021/03/05 04:06 AM
                    memory errorsLinus Torvalds2021/03/05 11:59 AM
                      memory errorsrwessel2021/03/05 12:32 PM
                        memory errorsrwessel2021/03/05 12:37 PM
                        memory errorszArchJon2021/03/06 08:39 PM
                      memory errorsGabriele Svelto2021/03/06 12:58 PM
                  memory errorsJörn Engel2021/03/05 10:12 AM
                Amiga recoverable RAM disk?Carson2021/03/05 03:03 AM
                  Thanks - TIL a cool Amiga feature (nt) (NT)John2021/03/05 12:51 PM
                    Another cool Amiga feature, datatypesCharles2021/03/06 12:01 AM
                      Another cool Amiga feature, datatypesJukka Larja2021/03/06 01:23 AM
                      Another cool Amiga feature, datatypesAnon2021/03/06 12:40 PM
                      Another cool Amiga feature, filesystemsMarcus2021/03/07 12:28 AM
  CPU & Memory bit flipszArchJon2021/03/04 06:39 AM
    CPU & Memory bit flipsdmcq2021/03/04 06:59 AM
      CPU & Memory bit flipsrwessel2021/03/04 12:27 PM
  speak of the devilRobert Williams2021/03/05 07:53 AM
    speak of the devildmcq2021/03/05 11:26 AM
      speak of the devilRobert Williams2021/03/05 03:15 PM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell avocado?