slacker (s@lack.er) on 1/27/09 wrote:
>Linus Torvalds (torvalds@linux-foundation.org) on 1/27/09 wrote:
>>slacker (s@lack.er) on 1/27/09 wrote:
>>>
>>>What if you have 5 applications wanting to write metadata
>>>about a file? Keep 5 textfiles around in a directory?
>>
>>No, keep their own damn databases, and index it by the
>>filename or something. Different programs should just
>>keep their data separate. They shouldn't mix it up with
>>files that they don't control.
>
>1176 words, and you start with this. If the application is
>indexing the files by name, what happens when the name
>changes?

You really don't understand the problems, do you?

I very much mentioned the case of the name changing later
in the post when I talked more about using a database
instead:

>>Again, it may appear slightly less convenient ("But but
>>but .. I _want_ my annotations to move around with the
>>file, and my brain is too small to think about security
>>issues"), but again, it's actually not a totally broken
>>model.

but you didn't seem to react to that part.

I acknowledge that it's "convenient" to have the metadata
follow a rename, but you don't seem to be able to fathom
why it's such an idiotic idea. The whole "convenience"
factor is simply not compelling enough, not when you
actually understand the problems.

And I tried to outline at least part of the problems. But
it all basically boils down to the fact that you don't
know, and fundamentally cannot know, whether the
metadata actually does follow the data, and whether it was
intentional or not.

(And this is what the security problems really boil down to
as well - the "don't know" and the "intentional or not"
part are both huge security issues).

For example, does it follow the data when you do a "mv"?

Here's a simple fact: you can never know. A trivial
file rename may well strip the data off the file. Do you
really want that kind of random behavior to be part of your
magic metadata? Hint: think cross-filesystem moves that
turn into copy/delete operations. And realize that you as
a user don't generally even know where the filesystem
boundaries are.

Or how about this: is your meta-data even valid after the
file got moved or copied and perhaps given to another
person? What about after changes?

Another question: say that you do edit the file.
Did you realize that 99% of all editors will have thrown
away your precious resources, because the way they edit
files is to create a new one, and then rename it over the
old one. But other changes to the file may not do that,
and may edit the file in place.

A fourth one: do you expect the meta-data to work when you
access the same file across the network, say over NFS?

Good luck with all that.

In other words: yes, extended attributes may be "convenient"
but they are totally an utterly broken. You will simply
never get any kind of "repeatable" behaviour. That's
simply pretty fundamental to the whole notion.

In contrast, using a database that your indexing program
controls at least gives you unequivocal answers to all the
above questions. It may not be as convenient (put another
way: the answers may not be the ones you want), but
it at least has well-defined semantics.

This is what I mean by "fundamentally broken". EA's give
you random associations. They're not reliable. They are
only reliable if you have strict rules for the file in
question, but if you have those then you're really much
much better off by just making that part of the
file format instead, and moving the metadata into
the file itself.

You can't have it both ways. Either you control the files
(in which case EA's are not the best answer), or you don't
(in which case EA's simply don't work reliably anyway).

And if you're willing to do the crap-shoot approach (the
"I don't need strict semantics, just a best-effort thing"
kind of situation where you don't really control the file,
but you feel confident enough that you're willing to gamble)
then you're still back to "a separate index over the
filename and/or inode/device number works better and more
reliably" case.

You can ask "what if the filename changes" all you like,
but you're ignoring the fact that EA's don't handle that
case any better anyway.

Linus