Hi,

nick (anon@anon.com) on 5/16/06 wrote:
>Brendan (btrotter@gmail.com) on 5/16/06 wrote:
>>nick (anon@anon.com) on 5/15/06 wrote:
>
>>>Do you use threads of a single memory space running on
>>>different nodes?
>>
>>Yes, but I split user space into "process space" and "thread space", such that
>>thread space can't be accessed from other threads. It's a little like "thread local
>>data" in POSIX, only implemented so that seperation is enforced. The disadvantage
>>is that switching between threads that belong to the same process involves changing
>>address spaces and is as expensive as switching between
>processes. The advantages
>>are that security of the threads local data is enforced, a thread's data doesn't
>>suffer from cacheline bouncing (or "across NUMA node" access penalties if the process
>>itself isn't tied to a specific NUMA node), the linear
>
>So you migrate the page when the thread moves across CPUs?

Each process has CPU affinity mask and each thread has a CPU affinity, where the process's CPU affinity mask is used to restrict the thread's CPU affinity (and the thread's CPU affinity determines which CPUs it can run on). Normally, when a process is created it's CPU affinity mask is set to all CPUs within the least-loaded NUMA node, and each thread's CPU affinity is set to a subset of this. These CPU affinities can be changed, but normally aren't.

The code that determines which pages are "good canditates" for sending to swap needs to track page usage - I'm hoping to re-use this to determine which physical pages are worth replacing when they belong to the wrong NUMA domain for the thread's CPU affinity (for thread space) or the wrong NUMA domain for the process's CPU affinity mask (for process space).

This avoids replacing all pages when a thread or process is migrated, but means that performance will be worse immediately after migration and will improve with time. It also means that pages that are good candidates to send to swap won't be replaced (unless they actually are sent to swap and then loaded back in).

It also takes care of the problem caused when a thread/process needs to allocate more RAM but there is no free pages for a specific NUMA domain. In this case a "less than perfect" page would be allocated, and would be replaced by a better physical page later on when better physical pages become free.

It's something that will need a lot of benchmarking and tuning to get right though...

>You still have program text and the wider memory space
>bouncing though.

Yes.

>AFAIK there has been work in the past on Linux (and probably
>other OSes) to do things like NUMA page migration, and even
>pagecache NUMA replication (for unmapped or readonly mapped
>pages). Not sure the state of these.

You may mean this patch: http://lwn.net/Articles/179934/

>>>Wrong. Number of cores has nothing to do with it, and
>>>desktops/workstations/small servers will never care much
>>>about NUMA issues because there just aren't enough sockets
>>>to make a difference. Improvement on even an 8 socket
>>>Opteron is probably unmeasurable on Linux, for example.
>>
>>For Opteron one hop is about 25% slower and 2 hops is about 50% slower. I couldn't
>>find figures for 3 hops (which is necessary for 8 sockets when there's only 3 hypertransport
>>links and something needs to connect to an I/O hub), and the figures I did find
>>vary a fair bit between different sources.
>
>I didn't mean memory latency, obviously that is easily
>measurable and relevant to real workloads. I was talking
>about kernel text replication. icache is mostly very well
>behaved (readonly, good locality, high frequency of use)
>and pretty easy to prefetch.

Does the kernel have a ".data" section?

>Well good luck with it. Sounds fun -- post a link if/when
>you feel it is a bit more interesting.

Thanks - don't worry, when I feel it's interesting I'll be plastering links everywhere I go... :-)


Cheers,

Brendan