Howard Chu (hyc.delete@this.symas.com) on January 9, 2020 10:39 pm wrote:
>
> Got a link to a bug report for gmake? Has it been fixed yet?

The really sad part is that it was fixed over two years ago, but GNU make hasn't had a real release since.

So the fix has been in the GNU make sources since mid-2017, and had this fix:

[SV 51159] Use a non-blocking read with pselect to avoid hangs.

but because most distributions (quite reasonably) will not take tooling updates without proper releases, almost all of them had the bug still because there hasn't been a new full release of GNU make in a long time.

> Fwiw, I invented that jobserver code back in 1991 while I was working at JPL,
> and back then there was no consistent shared memory API in Unix, but pipe()
> existed everywhere. And the default pipe buffer size was 4096 bytes, so it
> even worked fine on our 1024 node Connection Machine. (I recall a gmake
> bug report where it deadlocked on a system with more than 4096 CPUs, a few years
> back.) Overall, not a bad thing for a concept to have survived for the past 29
> years across multiple POSIX OSs.

Oh, the concept is perfectly fine.

And maybe even the original implementation was fine.

But at some point, the jobserver code then got a race where the following can happen:

- the pselect() says that the pipe is readable, so there is at least one token

- but in the meantime another sub-make comes along and steals the very last token

- but because pselect returned "readable", the other jobserver user does the read() and now blocks because all the tokens are gone again.

- before a new token comes in, a jobserver child job exits, and now because the sub-make is blocking for reads (and because the jobserver blocks SIGCHILD in general outside of the pselect), it doesn't react, so it won't release the token that the child job held.

End result: with just the right timing, you basically end up losing tokens temporarily. And on at least some machines it's really easy to then trigger just the right timing. And it's so easy to trigger, that you actually can end up basically losing all tokens, and the jobserver goes into pretty much a single-threaded mode where there originally was a large number of tokens, but every token but one is held by a user that is about to free one, but is waiting for a new token before it can do so.

And they get woken up one by one as that last token gets released, and at that point they then release their own token, so things make progress, but instead of having a build do 32 jobs at a time, you end up having a build that does one or two jobs at a time.

And what happened was that as we were cleaning up the pipe write-to-read path and in particular when it was made to be directed (so that a writer that wrote one byte woke up exactly one target reader), it was much easier to hit that particular case: everybody got woken up by "pselect()" (not exclusive), but obviously only one actual reader got the token.

So the exact timing with "lots of other readers end up blocking" happened quite reliably.

The fix is to make the read be non-blocking. As it pretty much always is when you use select() or poll(). Doing a select() on a blocking file descriptor is almost a bug by definition.

It was "fun" to debug this. I was obviously working on the pipe code, and obviously blamed the kernel when my build times got much worse. Let's just say that I spent a fair amount of time staring at everything pipe-related in the kernel and getting quite frustrated ;)

Linus