As always I agree with the *vast* majority of what Linus says, so I'll take the liberty of just snipping out the portions where I diverge.

Linus Torvalds (torvalds.delete@this.linux-foundation.org) on November 21, 2013 3:12 pm wrote:
> Amiba Gelos (b98202036.delete@this.ntu.edu.tw) on November 19, 2013 8:36 pm wrote:
> >
> >
> > Second, I wonder whether the claim that Knights Landing is bootable really implies we need good scalar
> > core. A major advantage of bootable accelerator is that by
> > moving from a heterogeneous MP to SMP the programming
> > paradigm is greatly simplified. Therefore, faster scalar execution is not really needed.
>
> People definitely need faster scalar execution.
>
> Guys, don't look at just DGEMM benchmarks or pure "throughput" stuff. The real HPC
> market cares deeply about things like "sufficient NFS client performance" for example,
> because in reality, that's how most of those clusters end up doing a lot of the data
> distribution.

Most real HPC systems have dedicated I/O nodes for that sort of thing, whether it be the Xeon/Opteron in "Xeon/Opteron + Tesla/Phi", or dedicated cores in BG/Q systems. That isn't really a reason to constrain the architecture of the *compute* nodes.

> Prediction: the whole PCIe card model will die, very much due to this kind of
> pressure.

The reason why the PCIe card model exists at all is economics: Discrete PCIe GPUs were being designed anyway for a much more lucrative market, and the HPC guys got to go along for the ride (albeit at the cost of the development hassles that you describe).

I therefore think that its death or lack thereof will be very much a function of how the graphics market evolves. If discrete GPUs die (or at least become such a small niche that the economies of scale vanish) then I think you'll be prove right, but not for the reasons you cite.