MoTheG (better@not.tell) on 11/12/09 wrote:
---------------------------
>? (0xe2.0x9a.0x9b@gmail.com) on 11/12/09 wrote:
>---------------------------
>>Couldn't that (at least partially) be solved by inserting a buffer between the pipeline stages? Like:
>>
>>[parser thread] ---> [buffer] ---> [compiler thread]
>
>I imagined this less strictly structured, I just though have one thread keep parsing
>all the files to the structure the compiler will use and let the compiler(s) handle
>them at the rate it or them get(s) there.
>If the parser is significantly faster than all the rest, it should pause and wait.
>I don't see why the parser should work just on demand, is the RAM that limiting?

Like I said, sometimes the parser is slower than the code generator. For example, when using recursive C++ templates.

It sounds like you guys are talking about work division and synchronization, which is important in any multithreaded program. However, on-demand, highly nondeterministic processing tends to be bad for caches, making it a bad strategy for compiler organization. You might even be better off exploiting code and data locality in a single slow thread than trying to saturate the resources of a multicore chip in a single process. If things work efficiently in a single thread, they should (though it's not a guarantee) remain so when multithreading is accomplished by multiple processes.

Really, I think most programmers are OK with breaking their source into files which are conveniently sized for both themselves and the compiler. Parallel make and precompiled headers will reliably perform pretty well.