Big gotchas: poor specs

By: Linus Torvalds (torvalds.delete@this.linux-foundation.org), August 13, 2019 11:31 am
Room: Moderated Discussions
aaron spink (aaronspink.delete@this.notearthlink.net) on August 12, 2019 9:45 am wrote:
>
> Program compilation scales even better if you don't do it on your client.

That's not necessarily true.

It really depends on the situation.

A build farm makes sense for things that take hours to build. The traditional "make world" kind of workflow, or for things where you do hundreds (or thousands) of builds for lots of different configurations.

A build farm also makes sense if you want to have weak clients for whatever reason (usually price).

But a build farm quite often loses out - and possibly badly - to a single powerful machine for a dedicated developer.

Just as an example: for the "nothing much changed" cases when you rebuild just a file or two incrementally, my local machine does a full ("allmodconfig") build in 30 seconds. That's mostly just "make" doing a lot of string processing (the kernel build is based on lots of clever rules that are very make-string-intensive), along with about 40 million "stat()" calls.

Those 40 million stat() calls are a non-issue if you have a local filesystem that is cached, and a good OS that does well on filename lookup. The string handling in "make" dominates. But just those 40 million stat calls in 30s would be "exciting" in a distributed network filesystem for a farm.

You simply will not be beating that with a cheap farm. Can you beat it with top-of-the-line dedicated resources? Of course. But then we'd not be talking your simply standard build farm any more.

Just checking out the tree on a farm might take longer than the local incremental build, and if you're doing some network filesystem from a single point just distributing it all will still be more expensive. So to beat it, you'd basically have to have a live filesystem either already replicated on the farm or a single very beefy machine indeed that is 5x faster than the powerful workstation.

Of course, then somebody changes a core header file, and my "30 seconds" becomes "15 minutes". At which point even a standard farm could beat it without lots of magic, but it's why I'm expecting to upgrade my machine next year - to make even the worst case be single-digit minutes.

But yes, this is the "developer who gets a top workstation" case. Enough RAM, good SSD disk and basically top-of-the line workstation CPU.

Can you beat it with very specialized build environments? Sure. If you replace "make" with something that is inherently git-aware and doesn't need to check out the whole tree and is smart enough to use git hashes instead of "stat" calls to track dependencies, and has fancy replicated caches for previous builds so that you can do optimal incremental builds etc.

But build systems are actually really complicated things, and there are lots and lots of subtle issues. For the kernel, we have rather complicated helper 'Makefile' rules that then make all the individual subdirectories have very simple and powerful build rules. And we have spent years on a very kernel-specific configuration language etc, but nobody has ever been willing to really spend years (or decades) on a 'make' replacement.

So there are things that in theory could scale really well in a distributed model, but building projects with 'GNU make' is not one of them. And while I've seen lots of fancy make replacements, people invariably tend to hate them unless they match their _exact_ situation.

A local development machine has some huge inherent advantages.

Of course, it does mean that I try very hard to avoid doing my merge windows (when I get a lot of those "full rebuild" cases) on the road. Then I just have a (good) laptop, and things are different. I don't tend to do full builds all the time at that point, and rely more on all the test infrastructure we do have that is done by various automated build farms.

But having a powerful workstation to do basic testing before then pushing out into the public is a good thing. The distributed test automation will catch the odd cases, but I want to be able to do basic sanity testing on my own first.

Linus
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
EPYC SAP-SD 2-tier benchmark resultsMichael S2019/08/08 05:18 AM
  EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownMichael S2019/08/08 08:29 AM
    Wrong headline. Should be "SPECpower_ssj2008 - EPYC2 is in class of its own" (NT)Michael S2019/08/08 08:51 AM
    EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownAlberto2019/08/08 03:42 PM
      EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownblue2019/08/08 07:35 PM
      EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownMaynard Handley2019/08/08 08:58 PM
        EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownanon2019/08/09 02:00 AM
          EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownAlberto2019/08/09 03:09 AM
            EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownblue2019/08/09 08:47 AM
        EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownAlberto2019/08/09 02:12 AM
          EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownMaynard Handley2019/08/09 09:25 AM
            EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownDoug S2019/08/09 09:33 AM
              EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownanon2019/08/09 10:24 AM
            EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownAlberto2019/08/10 04:04 AM
          EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownanon.12019/08/09 05:30 PM
        EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownme2019/08/09 09:58 PM
      EPYC SAP-SD 2-tier benchmark results - EPYC2 is in class of its ownanon2019/08/09 12:47 AM
  EPYC SAP-SD 2-tier benchmark resultsLinus Torvalds2019/08/09 03:29 PM
    EPYC SAP-SD 2-tier benchmark resultsanon2019/08/10 07:20 AM
      EPYC SAP-SD 2-tier benchmark resultsme2019/08/10 07:36 AM
        EPYC SAP-SD 2-tier benchmark resultsanon2019/08/10 08:20 AM
          EPYC SAP-SD 2-tier benchmark resultsanon.12019/08/10 09:40 AM
            EPYC SAP-SD 2-tier benchmark resultsanon2019/08/10 10:38 AM
          EPYC SAP-SD 2-tier benchmark resultsblue2019/08/10 09:47 AM
        EPYC SAP-SD 2-tier benchmark resultsAdrian2019/08/10 10:43 AM
          EPYC SAP-SD 2-tier benchmark resultsme2019/08/10 11:06 AM
            EPYC SAP-SD 2-tier benchmark resultsMichael S2019/08/10 11:28 AM
    Big gotchas: poor specsjohn2019/08/10 10:54 AM
      Big gotchas: poor specsme2019/08/10 11:04 AM
      Big gotchas: poor specsanon2019/08/10 11:25 AM
        Big gotchas: poor specsLinus Torvalds2019/08/10 12:13 PM
          Big gotchas: poor specsanon2019/08/10 01:58 PM
          Big gotchas: poor caches2019/08/12 01:51 AM
            Big gotchas: poor caches2019/08/12 01:59 AM
            Big gotchas: poor cachesTravis Downs2019/08/12 09:15 AM
              parallelism vs. efficiencyhobold2019/08/13 04:43 AM
            Big gotchas: poor cachesAndy N2019/08/12 06:45 PM
              Big gotchas: poor cachesanon2019/08/13 12:43 AM
          Big gotchas: poor specsJouni Osmala2019/08/12 10:18 PM
            Big gotchas: poor specsAdrian2019/08/12 11:58 PM
              Kernel compile 25-30s, noise 25-27dBJouni Osmala2019/08/13 04:00 AM
                Quiet and fast workstationsGian-Carlo Pascutto2019/08/13 06:00 AM
                  Quiet and fast workstationsAdrian2019/08/13 07:01 AM
                  Quiet and fast workstationsDavid Hess2019/08/13 01:51 PM
                    Quiet and fast workstationsGian-Carlo Pascutto2019/08/14 12:43 AM
                      Quiet and fast workstationsMichael S2019/08/14 02:09 AM
                        Quiet and fast workstationsGian-Carlo Pascutto2019/08/14 02:59 AM
                          Quiet and fast workstationsDavid Hess2019/08/14 06:05 PM
                        Quiet and fast workstationsDavid Hess2019/08/14 05:56 PM
                      Quiet and fast workstationsDavid Hess2019/08/14 06:30 PM
                  Passively cooled threadripper/xeon WJouni Osmala2019/08/15 03:12 AM
                    Passively cooled threadripper/xeon WAdrian2019/08/15 03:47 AM
                      Passively cooled threadripper/xeon WJouni Osmala2019/08/15 05:58 AM
                        Passively cooled threadripper/xeon WAdrian2019/08/16 12:12 AM
                          Passively cooled threadripper/xeon WAdrian2019/08/16 12:21 AM
                          Passively cooled threadripper/xeon WJouni osmala2019/08/16 09:46 AM
                            Passively cooled threadripper/xeon WGroo2019/08/16 12:07 PM
                Kernel compile 25-30s, noise 25-27dBAdrian2019/08/13 06:39 AM
                Kernel compile 25-30s, noise 25-27dBLinus Torvalds2019/08/13 12:12 PM
                  which graphics card?anonymous22019/08/13 12:52 PM
                    which graphics card?Linus Torvalds2019/08/13 01:12 PM
                  Kernel compile 25-30s, noise 25-27dBDoug S2019/08/13 03:22 PM
                    Kernel compile 25-30s, noise 25-27dBDavid Hess2019/08/14 06:38 PM
                  Kernel compile 25-30s, noise 25-27dBJouni Osmala2019/08/14 12:09 AM
                    AIOsGroo2019/08/14 07:19 AM
                      AIOsGian-Carlo Pascutto2019/08/14 08:52 AM
                        AIOsGroo2019/08/14 09:49 AM
                      AIOsBrett2019/08/14 02:02 PM
                    Kernel compile 25-30s, noise 25-27dBLinus Torvalds2019/08/14 11:10 AM
                  Kernel compile 25-30s, noise 25-27dBGian-Carlo Pascutto2019/08/14 01:33 AM
                    Kernel compile 25-30s, noise 25-27dBAdrian2019/08/14 05:06 AM
                      Kernel compile 25-30s, noise 25-27dBGian-Carlo Pascutto2019/08/14 09:01 AM
                      AMD Bios, etc..Anon2019/08/14 11:39 PM
                        AMD Bios, etc..hobold2019/08/15 07:10 AM
                          AMD Bios, etc..Anon2019/08/15 04:20 PM
                        AMD Bios, etc..Jan Olšan2019/08/15 08:49 AM
                          AMD Bios, etc..Adrian2019/08/15 09:16 AM
                          AMD Bios, etc..Gro2019/08/15 10:57 AM
                        AMD Bios, etc..David Hess2019/08/16 05:39 AM
      Big gotchas: poor specsAdrian2019/08/10 01:28 PM
      Big gotchas: poor specsDummond D. Slow2019/08/10 01:57 PM
        Big gotchas: poor specsanon22019/08/11 12:33 AM
          Big gotchas: poor specsblue2019/08/11 12:57 AM
          Big gotchas: poor specsAdrian2019/08/11 05:41 AM
            Big gotchas: poor specsaaron spink2019/08/12 09:45 AM
              Big gotchas: poor specsFoo_2019/08/12 11:02 AM
              Big gotchas: poor specsTravis Downs2019/08/12 11:26 AM
                Big gotchas: poor specsGabriele Svelto2019/08/13 09:48 AM
                Big gotchas: poor specsaaron spink2019/08/13 07:23 PM
                  Big gotchas: poor specsDoug S2019/08/13 11:15 PM
                    Big gotchas: poor specsGian-Carlo Pascutto2019/08/14 01:35 AM
                    Big gotchas: poor specsaaron spink2019/08/15 05:16 PM
                      Big gotchas: poor specsJouni Osmala2019/08/15 09:03 PM
                        Big gotchas: poor specsaaron spink2019/08/19 04:05 AM
                          Big gotchas: poor specsMichael S2019/08/19 04:37 AM
                            Big gotchas: poor specsRicardo B2019/08/19 05:40 PM
                              Big gotchas: poor specsAaron Spink2019/08/25 02:29 AM
              Big gotchas: poor specsAdrian2019/08/12 11:31 AM
              Big gotchas: poor specsJukka Larja2019/08/13 07:37 AM
              Big gotchas: poor specsLinus Torvalds2019/08/13 11:31 AM
          Big gotchas: poor specsDummond D. Slow2019/08/11 06:23 AM
      FX-9590 > i7-9700KGian-Carlo Pascutto2019/08/12 10:37 AM
    RDRAND issueanonymou52019/08/10 02:52 PM
      RDRAND issueLinus Torvalds2019/08/10 04:15 PM
        RDRAND issueGionatan Danti2019/08/11 01:35 AM
          RDRAND issueLinus Torvalds2019/08/11 10:43 AM
            RDRAND issueLinus Torvalds2019/08/11 11:05 AM
              RDRAND issue@never_released2019/08/11 11:10 AM
                RDRAND issueLinus Torvalds2019/08/11 01:13 PM
                  Err....Groo2019/08/11 06:47 PM
                    Err.... This is a zero content fud posteye roll2019/08/11 08:48 PM
                    Err....Brendan2019/08/11 08:50 PM
                    Err....Jukka Larja2019/08/12 07:38 AM
                  RDRAND issueJose2019/08/12 02:33 AM
              RDRAND issueGionatan Danti2019/08/11 02:12 PM
            RDRAND issueaaron spink2019/08/12 09:18 AM
      RDRAND issuename2019/08/11 09:35 AM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell avocado?