Brett (ggtgp.delete@this.yahoo.com) on July 18, 2022 12:38 pm wrote:
> Ben T (ben.delete@this.noemail.com) on July 16, 2022 4:50 pm wrote:
> > >> Would you say even the Last Level Cache or System Level
> > Cache should not be shared with an untrusted user?
> > >
> > > Yes, certainly. And not only that. Even memory and disks
> > > shouldn't be shared. IMHO, even SAN shouldn't be shared.
> >
> > Do people here think the ideal public cloud infrastructure would be a cluster in a box rather than a single
> > symmetric multiprocessor running virtual machines from multiple
> > users? For example, imagine a 1U chassis containing
> > 12 processor chips and 12 hot-swap 2.5” U.2 NVMe drives. Each processor chip could include:
> >
> > 8 performance CPU cores plus caches
> > interfaces to two 128-bit LPDDR DRAMs
> > 4 PCIe lanes to connect to one NVMe drive
> > a 10G ethernet connection
> >
> > If the minimum unit a cloud user could get is 8 cores, they would not be sharing this chip
> > with anyone else so the risk of data leakage between users would be greatly reduced. If
> > it is possible to combine these processor units for bigger single tasks but still keep
> > them partitionable into independent units for small tasks, that would be even better.
>
> A 12 socket cluster in a box is far better than doing a 12 socket unified memory system.
> Linux would be allocating memory two sockets away on the mesh and performance would go strait
> to hell. You would need a ridiculous number of PCIe channels in a mesh to try and overcome
> the bottlenecks by brute force, and pay a needless heat cost for that connect.

If a job had to allocate memory on another socket then a cluster will also be horrible. Possibly even worse. What are you even talking about? A cluster doesn't magically make inter-chip communication free. "just add a switch then communication does not matter" -- this is nonsense.

>
> With cluster in a box you can use desktop/laptop chips and at most add a network switch to both
> share the networking and communicate between the sockets without hitting the Ethernet.
>
> This is the future for 1U server boxes. But Intel and AMD have financial and competitive
> reasons to keep severs using DIMM’s and unified memory, moat maintenance reasons.
>
> I expect the new server entrants to copy Apple with RAM on socket over the next 2-5
> years, and only after loosing market share for Intel and AMD to release such sockets.