? (0xe2.0x9a.0x9b@gmail.com) on 9/7/09 wrote:
---------------------------
>megol (golem960@gmail.com) on 9/6/09 wrote:
>---------------------------
>>? (0xe2.0x9a.0x9b@gmail.com) on 9/6/09 wrote:
>>---------------------------
>>>anon (anon@anon.com) on 9/6/09 wrote:
>>>---------------------------
>>>>[cut]
>>>>So I just think that seperate I+D caches are used for other reasons (as you now
>>>>say above), and the real way to get cache efficiency in streaming workloads is to
>>>>have cache control instructions and attributes (nontemporal, uncached, etc). Especially
>>>>if you have subsequent cache levels unified you don't want useless streaming data to pollute them either.
>>>
>>>I think that cache-hinting instructions are a bit controversial in the sense that
>>>they are not future-proof. Compare the following:
>>>
>>>Year X: A typical processor has N megabytes of data-cache (nowadays N=2..6 or something).
>>>
>>>Year X+5: A typical processor has some 5*N megabytes of data-cache.
>>>
>>>In case you run your program in year X (=now), it may *require* usage of cache-hinting
>>>instructions in order to run optimally - because the data-set does not fit into
>>>the cache, so the data-set is polluting the cache.
>>>
>>>However, when you will be running the program in year X+5, the data-set may well
>>>fit into the data-cache - so the program is no longer polluting the cache. However,
>>>and this is The problem here, because your program uses cache-hinting instructions
>>>the data which would have normally stayed in the cache is being downloaded/uploaded to/from memory.
>>>
>>>So, if in the year X+5 you remove the cache-hinting instructions from your program,
>>>the program might (I am not saying that it will, might!=will) run substantially
>>>faster without the cache-hinting instructions.
>>>
>>>In my opinion, the fact is that most programs do not use cache-hinting instructions
>>>at all. And this is good - because otherwise they may run sub-optimally on future
>>>processors with larger caches. So in a sense, the *absence* of cache-hinting instructions
>>>ensures that a program will scale with the cache size.
>>>
>>So what stops this future processor to either ignore the hints altogether or do
>>a cache tag lookup and ignore hints that are already resident in cache?
>
>Who/what is going to decide whether to ignore the hints? This has to be a conscious
>process of answering the question "Why are we designing a processor that is going to ignore the hints?".
>
>AMD documentation on 10h processors says:
>
>"The PREFETCHNTA instruction performs a nontemporal prefetch. The data is read
>into the L1 data cache; to avoid cache pollution, when a PREFETCHNTA misses in the
>L2 cache and reads from memory, the data is never evicted to the L2 cache. When
>a PREFETCHNTA hits in the L2 cache, the data is evicted back to the L2 cache."
>
>So a scenario which is not future-proof might look like this:
>
>1. The program is reading an array. Assume the array is being read from memory
>(do not ask how it did happen that the data is not in any cache - simply assume it is happening this way).
>
>2. The array is accessed sequentially and as a whole does not fit into the cache.
>Therefore, it seems like a good idea to use the PREFETCHNTA instruction. After an
>array element has been processed, it is later evicted from L1 and does not go into L2.
>
>3. After processing the array, another part of the program executed sometime later
>will be working with the very same array.
>
>Let's compare years X and (X+5) and (X+10):
>
>Year X: The program works optimally and as expected.
>
>Year (X+5): L2 cache is larger. Step 3 is reading the array data from memory (or
>alternatively from L3 cache), but a *closer examination* will show that the data
>fits into L2 cache so there is no reason to evict it from L2 into memory (or L3 cache).
>
>Year (X+10): L1 cache is so large that the whole dataset now fits into L1 cache.
>Thus the array is not evicted from L1. Therefore step 3 now runs with blazing speed when compared to year (X+5).
>
>Note1: In both X and (X+5), the array does not fit into L1 cache.
>
>Note2: As I wrote above, the existence of the problem is only revealed after "closer examination".
>
>Note3: Instead of years X, (X+5) and (X+10) you can easily imagine customers A,
>B and C with different machines. Customer B is reporting to you that the program
>runs only slightly faster than on A's machine, despite the fact that B's machine
>is more expensive and is supposed to be noticeably faster.

Really, something that works well if it fits in cache but badly if it's not (ie. most things that will reuse the data in a relatively close period of time) should be using cache blocking techniques as much as possible unless the array is trivially sized to fit in cache.

Something that knows it will not be used again should use streaming access hints.

Of course, nether of these is always possible in all cases, but then the algorithm, with or without such hints, is always going to benefit with some manual tuning I'd say. Even in year X+N, people will have different lines of CPUs and ones from different vendors, so cache hierarchies are going to be different.

I don't think it is too much of a burden on the programmer to be expected to do a bit of tweaking to make the most out of the cache hierarchy in critical routines.