AlbertH (please.delete@this.email.address) on July 10, 2013 4:50 pm wrote:

After Haswell-E (maybe Broadwell-E), I think Intel will embed Crystalwell cache chips
in the printed circuit board of the package substrate. They could put 4 Crystalwell
chips on the 4 sides of a processor in the epoxy of the package substrate and 4 more
Crystalwell chips on top of the package substrate. To do this, they would solder 4
Crystalwell chips to the bottom half of the package substrate. Then they would put
a layer of epoxy and glass fiber on top of the 4 chips between the bottom half of the
package substrate and the top half of the package substrate. Then they would heat and
press this sandwich. Then they would drill and plate the vias. Then they would test
the assembly. Finally, they would solder the processor and the top 4 Crystalwell chips
to the top of the package substrate. This would provide 1 or 2 GBytes of cache using
128 MByte or 256 MByte Crystalwell chips. I expect this to first appear in the 4-socket
Xeons. This 1 or 2 GBytes will be the last level of cache. There could be one or two
smaller and faster cache chips on top of the processor in addition to this last level
of cache. Part of the L3 cache in the processor could be repurposed to hold the tags for
this enormous last level cache. Examples of applications for this type of assembly are
in-memory databases such as SAP HANA and bioinformatics. This assembly could also be
used in supercomputers with no additional DRAM. I don't have any confidential information
so this is all speculation on my part.

Instead of putting the cache chips on 4 sides of the processor chip, another way to do this is to thin the cache chips and stack the cache chips on top of each other. Stacking the cache chips would result in a smaller package so the lower-end processors would not be burdened by a large package to make the processors interchangeable. Up to 8 flash memory chips are regularly stacked on top of each other today.

Through silicon vias would be needed because of the wide interface between the cache chips and the processor chip. Through silicon vias are said to be expensive but Intel currently charges up to $2500 per processor for their dual-socket server chips (Ivy Bridge-EP) and up to $6600 per processor for their quad-socket server chips (Ivy Bridge-EX). Through silicon vias are not expensive compared to the price of a multi-thousand dollar processor.