slacker (s@lack.er) on 1/27/09 wrote:
>
>What if you have 5 applications wanting to write metadata
>about a file? Keep 5 textfiles around in a directory?

No, keep their own damn databases, and index it by the
filename or something. Different programs should just
keep their data separate. They shouldn't mix it up with
files that they don't control.

That not only makes them portable (not just across different
OS's, but across filesystems on the same OS), but really,
using extended attributes for "tagging" files is broken.
It's horrible.

It's horribly broken for so many reasons, many of them
totally trivial. Security being one (and as a special case,
read-only media being an issue of security being done on
a media level). Performance being another. The whole data
portability being yet another. Or something as fundamental
as just uninstalling/reinstalling the app in question.

Yes, people keep wanting to do this, but no, it does not
become a "good idea" just because people continually make
the same fundamental mistake over and over again.

Take an example of the kind of brokenness it results in:
the horrible indexing thing that is used by "beagle" (one
of the search tools for the Linux desktop). It uses
extended attributes to hide its cookies (hashes of the
content), and it's a total disaster. And it's even
the recommended model.

What does it result in? Let me count the ways:

- totally horrid access patterns. Now the beagle data
is spread all over the disk, since any sane filesystem
will keep the extended attributes close to the file.

Sure, at least under Linux, accessing the data when it
is cached is a lot faster than going to some
external MySQL database, since the kernel basically does
a hell of a good job of doing filesystem lookups. So
the beagle guys apparently made the decision that it
should be the preferred model based on some benchmarks
where they probably had all the data in cache.

- It actually messes with peoples data.

Do you really think it's a good idea to change peoples
files as you index them and add your application-specific
turds of sh*t? Yes, extended attributes are generally
hidden, but they aren't invisible. Updating them changes
the inode times, which breaks things like backups and
good source control management.

- It's a bad idea in general, and for security in
particular.

What should a program do when it copies the file? Should
it copy the extended attributes? Should it not? What are
the security implications? When you copy a file to
somebody else, and you don't even realize that it has
some private data associated with it, did you just
intend to copy your own porn-filter annotations with it,
or not?

Hidden state (but see above how it's badly hidden)
is a security disaster. People transfer it (or not)
without realizing it, and that simply is not a good idea.

- In the same vein, what about multiple people sharing
access to that file? You may want a file to be readable
(or even writable) by a group, but what about the
extended attribute permissions? Should they be shared?
Unique? Writable separately?

- You always end up having to have another model for the
extended attributes anyway

To go back to the beagle example: beagle actually does
have a MySQL (or something) database as an alternative
model, since it wants to be able to index things that
are read-only, or exist on filesystems that do not
support extended attributes.

This is not a limitation of "some filesystems do
not support it". It's a very fundamental limitation of
the whole extended attribute mindset. Exactly the mindset
that you pointed to: "5 applications wanting to write
metadata to a text file".

The model is broken!

You want to write metadata for files you don't own, or
for files on a CD-ROM, or any number of examples like
that, where extended attributes simply don't work. And
since the whole point was that the applications in
question don't need to know about each other, and can
do these annotations independently, this is not going
away. There is never really a valid case where you can
say "this is a special application that always has
write access to the file on a filesystem that supports
extended attributes", because once you accept the notion
of a "special application" tie-in to the file, then you
damn well could have just made the whole file be app-
specific, and extended attributes bought you nothing
but grief.

I really have never ever seen a valid use of any
extended attributes. Yes, I've seen uses. But they've all
been broken (but "convenient") in some way. And if you
do not realize after my explanation how very broken your
example usage was, I can't really help you.

If you make explicit directories, all the above issues
go away. Yeah, it may not be as "convenient", and it may
look ugly. Tough. At least it's not a fundamentally broken
model.

The same goes for just doing your annotations in an app-
specific database. Again, it may appear slightly less
convenient ("But but but .. I _want_ my annotations to move
around with the file, and my brain is too small to think
about security issues"), but again, it's actually not a
totally broken model.

Can you use extended attributes sanely? Yeah, I guess doing
ACL's as extended attributes (and really making it part of
the system security model - ie they are not some kind of
per-app after-thought turd left around) is a sane model.
There are probably others, but it really is very hard to
see a huge point to them.

Now, just so that I don't sound like I'm purely negative:
I do understand some of the background for wanting the
extended attributes. People want to have "files", and they
hate having to open a filename like "directory/file" in
order to just group information together.

So I do understand that people say "I don't want to create
a whole new directory for all my data, just so that I can
then associate other things with that data".

That, as far as I'm concerned, seems to be the only
real reason why people want EA's in the first place, but
the thing is, EA's aren't really the right model. Exactly
because they have all the issues with the hidden state
being a bit too hidden (eg you can't have per-attribute
security rules), yet often then a bit too visible (ctime on
the main file changes when you change an attribute).

If you use directories, and then perhaps teach your file
tools to use the "data fork" (just a file with a special
name) of a directory implicitly (and that could even
involve teaching your OS kernel about them), you would get
most of those "usability" advantages of extended attributes
without the big downsides.

I dunno. But I do know that your particular example
is absolutely the worst possible argument for extended
attributes.

Linus