The fragility of your argument

By: Vincent Diepeveen (diep.delete@this.xs4all.nl), May 16, 2007 9:10 am
Room: Moderated Discussions
Linus Torvalds (torvalds@osdl.org) on 5/14/07 wrote:
---------------------------
>Dean Kent (dkent@realworldtech.com) on 5/14/07 wrote:
>>
>>It seems odd to me that virtually *every* address must
>>be mapped, whether actually used or not.
>
>Why would you ever have unused memory?
>
>Why did you buy such memory in the first place?
>
>Any sane OS (and yeah, XP/Vista may not fall under that
>heading) will basically always use all your memory, and
>you will never have anything unused.

huh?

You just reserve 32MB ram for shared memory, any allocation bigger than that needs root permission. Very ugly from user viewpoint.

Not many systems nowadays have just one core.

Can this get fixed (i remember in 1999 already Cox saying at the SMP list that this was not his fault, but now it's 2007 and still not fixed despite many people asking for a neat implementation)?

Also sharing memory with other processes, no matter how much more crappy it is under windows; sharing memory between processes in windows can easily slow you down factor 100+ nowadays, as it all gets allocated via the pagefile which is the most ugly way of doing it, and also takes care that if some DLL's hang around in RAM that your shared memory of a few gigabytes is dead slow, about a factor 1000 slower than you would want it to be; in linux it still gets done with private memory (why on planet earth can other processes read private memory of this process?), which is basically profitting from a bug in the implementation (otherwise sharing memory would not be possible). Just following the manual page, which lists also SHM_ANON (shared anonymousmemory) is just not implemented into the kernel.

It's incredible to me how both linux and windows are behind with respect to multiprocessing. Windows of course most of all. Nothing can save you in windows. Just nothing works there, especially not for memory that needs to get put in the same virtual adress space (mapviewoffileex). The only way to make a bigger chance of not allocating slow shared memory in windows is a regular reboot of the machine.

Now we didn't yet even touch the NUMA problem.

Of course past few years the kernels improved quite a bit there, yet not enough.

Running some long term calculations, where after some weeks or months 1 or 2 finish out of the 4 cores (dual opteron dual core) and not always immediately new processes start,

you just see how every few seconds the processes swap from processor to processor, based upon the load of the processor.

Very ugly.

Of course it's tough to check how many of those switches are switches to different memory controllers.

I feel however a few % more can be achieved there.

stuff like libnuma, it all isn't 100% working.

I want 0 flips of course.

Just my own private opinion...

Vincent

>Any unused memory is just wasted (pretty much by
>definition).
>So Linux will always use all your memory, even if it's
>just for caching (of course, "just" is unfair. Caching
>is just about the most important thing any OS can do, so
>it's definitely not a "just" thing).
>
>We keep a small percentage of memory free for emergency
>purposes, and if you've run a big process and exit it
>and throw it away we'll obviously temporarily have lots
>of free memory that we can't do much about, but in
>general you should never see any big amount of unused
>memory except just after bootup.
>
>Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Rock/Tukwila rumorsmas2007/05/05 11:59 AM
  Rock/Tukwila rumorsDavid Kanter2007/05/05 01:33 PM
    Rock/Tukwila rumorsDean Kent2007/05/05 02:35 PM
      K8 vs Win64 timelineanonymous2007/05/05 05:19 PM
        Yes, I misremembered...Dean Kent2007/05/05 09:03 PM
    RockDaniel Biz√≥2007/05/06 01:34 AM
      RockDean Kent2007/05/06 06:11 AM
    Rock/Tukwila rumorsJoe2007/05/06 10:24 AM
      Rock/Tukwila rumorsDean Kent2007/05/06 10:49 AM
      Rock/Tukwila rumorsLinus Torvalds2007/05/06 11:09 AM
      Rock/Tukwila rumorsanon2007/05/07 12:32 AM
        Rock/Tukwila rumorsRakesh Malik2007/05/07 08:36 AM
          Rock/Tukwila rumorsMichael S2007/05/07 09:06 AM
          Rock/Tukwila rumorsanon2007/05/07 08:48 PM
            Rock/Tukwila rumorsRakesh Malik2007/05/08 05:45 AM
              Rock/Tukwila rumorsanon2007/05/08 04:30 PM
                Wow. (nt)Brannon2007/05/08 05:16 PM
                Rock/Tukwila rumorsrwessel2007/05/08 08:48 PM
                  Rock/Tukwila rumorsJS2007/05/08 09:07 PM
                    Rock/Tukwila rumorsJS2007/05/09 05:44 AM
                Rock/Tukwila rumorsRakesh Malik2007/05/09 04:35 AM
                  Much ado about xMichael S2007/05/09 08:39 AM
                    Call it x86-64Linus Torvalds2007/05/09 09:27 AM
                      (i)AMD64Michael S2007/05/09 11:16 AM
                        (i)AMD64Linus Torvalds2007/05/09 11:29 AM
                          (i)AMD64Groo2007/05/09 03:45 PM
                          TIFNAAanonymous2007/05/09 04:49 PM
                            Inspired by FYR Macedonia? (NT)Michael S2007/05/09 10:21 PM
                              More likely...rwessel2007/05/09 11:39 PM
                            TIFNAAGabriele Svelto2007/05/09 10:57 PM
                          (i)AMD64James2007/05/10 01:27 AM
                        i86Dean Kent2007/05/09 11:30 AM
                        (i)AMD64Max2007/05/09 12:28 PM
                          wide86? long86?hobold2007/05/10 04:05 AM
                            x87 perhaps, it is one more. :) (NT)Groo2007/05/10 04:50 AM
                              x86+Dean Kent2007/05/10 07:44 AM
                                Does it really matter?Doug Siebert2007/05/10 08:10 AM
                                  let's stay with x86-64 for now, pleaseMarcin Niewiadomski2007/05/10 10:50 AM
                                    let's stay with x86-64 for now, pleaseDean Kent2007/05/11 05:11 AM
                                      let's stay with x86-64 for now, pleaserwessel2007/05/11 01:46 PM
                                        let's stay with x86-64 for now, pleaseDean Kent2007/05/11 05:03 PM
                                          let's stay with x86-64 for now, pleaseMichael S2007/05/12 09:49 AM
                                            let's stay with x86-64 for now, pleaseDean Kent2007/05/12 12:05 PM
                                              let's stay with x86-64 for now, pleaseMichael S2007/05/12 12:25 PM
                                                let's stay with x86-64 for now, pleaseDean Kent2007/05/12 02:39 PM
                                                  let's stay with x86-64 for now, pleaseJasonB2007/05/13 06:43 AM
                                                    client consolidationMichael S2007/05/13 07:37 AM
                                                  let's stay with x86-64 for now, pleaseTzvetan Mikov2007/05/13 02:44 PM
                                                let's stay with x86-64 for now, pleaserwessel2007/05/14 01:42 PM
                                      What's your point?Doug Siebert2007/05/11 01:56 PM
                                        What's your point?Linus Torvalds2007/05/11 03:15 PM
                                          What's your point?Doug Siebert2007/05/13 02:11 PM
                                            What's your point?Dean Kent2007/05/13 06:04 PM
                                              What's your point?JasonB2007/05/14 01:06 AM
                                                What's your point?Dean Kent2007/05/14 06:20 AM
                                                  What's your point?JasonB2007/05/14 03:35 PM
                                                  What's your point?JasonB2007/05/14 06:35 PM
                                                    What's your point?Dean Kent2007/05/14 07:12 PM
                                        What's your point?Dean Kent2007/05/11 05:06 PM
                                        What's your point?Stephen H2007/05/13 12:55 AM
                                          Why didn't MS take advantage of PAE?David W. Hess2007/05/13 07:37 AM
                                            PAE sucks (Why didn't MS take advantage of PAE?)Linus Torvalds2007/05/13 09:20 AM
                                              PAE sucks (Why didn't MS take advantage of PAE?)Dean Kent2007/05/13 09:49 AM
                                              PAE sucks (Why didn't MS take advantage of PAE?)David W. Hess2007/05/13 11:37 AM
                                              > 1 GB RAM on a 32-bit systemTzvetan Mikov2007/05/13 12:44 PM
                                                > 1 GB RAM on a 32-bit systemS. Rao2007/05/13 02:00 PM
                                                  > 1 GB RAM on a 32-bit systemTzvetan Mikov2007/05/13 04:32 PM
                                                    > 1 GB RAM on a 32-bit systemS. Rao2007/05/13 11:19 PM
                                                > 1 GB RAM on a 32-bit systemLinus Torvalds2007/05/13 02:46 PM
                                                  > 1 GB RAM on a 32-bit systemTzvetan Mikov2007/05/13 04:23 PM
                                                  > 1 GB RAM on a 32-bit systemJasonB2007/05/13 05:37 PM
                                                    Windows manages memory differentlyTzvetan Mikov2007/05/13 07:31 PM
                                                      Windows manages memory differentlyJasonB2007/05/14 12:50 AM
                                                        Windows manages memory differentlyTzvetan Mikov2007/05/14 07:56 AM
                                                          Windows manages memory differentlyrwessel2007/05/14 02:40 PM
                                                            Windows manages memory differentlyDavid W. Hess2007/05/14 03:07 PM
                                                              Windows manages memory differentlyrwessel2007/05/14 03:51 PM
                                                            Windows manages memory differentlyTzvetan Mikov2007/05/14 04:40 PM
                                                              Windows manages memory differentlyrwessel2007/05/14 05:09 PM
                                                      Windows manages memory differentlyHoward Chu2007/05/14 10:17 AM
                                                        Windows manages memory differentlyJukka Larja2007/05/14 10:30 AM
                                                        Windows manages memory differentlyTzvetan Mikov2007/05/14 12:54 PM
                                                          Windows manages memory differentlyHoward Chu2007/05/15 02:35 AM
                                                            Windows manages memory differentlyGroo2007/05/15 06:34 AM
                                                      Anyone know what OS X (10.4, Intel, desktop) does?Matt Sayler2007/05/15 05:23 AM
                                                        Anyone know what OS X (10.4, Intel, desktop) does?Wes Felter2007/05/15 07:37 AM
                                                        Anyone know what OS X (10.4, Intel, desktop) does?Anonymous2007/05/15 09:49 AM
                                                        Anyone know what OS X (10.4, Intel, desktop) does?anon22007/05/15 06:13 PM
                                              PAE sucks (Why didn't MS take advantage of PAE?)Paul2007/05/13 02:40 PM
                                                PAE sucks (Why didn't MS take advantage of PAE?)Peter Arremann2007/05/13 04:38 PM
                                                  PAE sucks (Why didn't MS take advantage of PAE?)Henrik S2007/05/14 02:31 AM
                                              The fragility of your argumentslacker2007/05/13 02:56 PM
                                                The fragility of your argumentnick2007/05/13 04:42 PM
                                                The fragility of your argumentHoward Chu2007/05/14 01:52 AM
                                                  The fragility of your argumentDean Kent2007/05/14 08:19 AM
                                                The fragility of your argumentanon22007/05/14 07:26 AM
                                                  The fragility of your argumentTzvetan Mikov2007/05/14 08:01 AM
                                                  The fragility of your argumentDean Kent2007/05/14 08:16 AM
                                                    The fragility of your argumentLinus Torvalds2007/05/14 10:57 AM
                                                      The fragility of your argumentJasonB2007/05/14 03:48 PM
                                                        The fragility of your argumentDean Kent2007/05/14 06:36 PM
                                                          The fragility of your argumentRicardo B2007/05/16 01:40 AM
                                                            The fragility of your argumentDean Kent2007/05/16 02:32 AM
                                                              The fragility of your argumentRicardo B2007/05/16 05:41 AM
                                                                PSRicardo B2007/05/16 05:50 AM
                                                                The fragility of your argumentDean Kent2007/05/16 08:07 AM
                                                                  Modern web browsingS. Rao2007/05/16 08:16 AM
                                                                    Aha!Dean Kent2007/05/16 08:27 AM
                                                                      Aha!Dean Kent2007/05/16 08:32 AM
                                                                        Aha!S. Rao2007/05/16 09:34 AM
                                                                  The fragility of your argumentRicardo B2007/05/16 09:00 AM
                                                      The fragility of your argumentVincent Diepeveen2007/05/16 09:10 AM
                                                        The fragility of your argumentPaul2007/05/16 02:01 PM
                                                          The fragility of your argumentVincent Diepeveen2007/05/17 02:05 AM
                                                    The fragility of your argumentanon22007/05/15 12:35 AM
                                                    Splits vs page allocations?Matt Sayler2007/05/15 06:33 AM
                                          What's your point?Michael S2007/05/13 07:55 AM
                                            What's your point?anonymous2007/05/13 10:08 AM
                                              What's your point?Michael S2007/05/13 10:31 AM
                                      let's stay with x86-64 for now, pleaseJasonB2007/05/13 06:16 AM
                                x864 =) (NT)some12007/05/15 02:03 AM
    Rock/Tukwila rumorsIntelUser20002007/05/06 01:27 PM
  Rock/Tukwila rumorsm2007/05/13 07:05 AM
  Rock/Tukwila rumorsmas2007/05/15 08:40 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell green?