Carson (carson.delete@this.example.edu) on September 16, 2021 1:12 pm wrote:
> anon (anon.delete@this.ymous.org) on September 14, 2021 3:17 am wrote:
> > I have one question regarding not doing a TLB check on each load though. What is the safety net when
> > you have an inflight store to VA A and an inflight load from VA B and A and B are aliased ?
> >
> > Let's assume you enforce only one alias in the L1D. If currently the resident alias is A, the load
> > will return L1D data and miss the dependency with the store (since A != B, and so far we are not
> > getting a translation for the load because it hit in L1D). If currently the resident alias is B,
> > then the load will miss and go get a translation, to realize that there is aliasing going on, and
> > then it is not obvious that it would not still get the L1D data rather than the store data.
> >
> > So, basically, how do you do memory disambiguation without having each load
> > get a translation just so it can compare to stores in the PA space ? I suppose
> > you could do a pessimistic check with just the set and offset bits ?
>
> I'm confuzzled. You say the store is to VA A and the load is from VA B. So
> if the resident alias is A, the load will miss in L1 and not return L1 data.
>
> It's also not totally clear what the program order of the two accesses is. I assume A is first, but if I start
> assuming you got something backward and try to infer what you really meant, that assumption gets shaky.
>
> Basically, the same problem arises in store buffers. If I have "store A", "store B", "load A",
> the load will be satisfied from the first store based on a VA match. If it later turns out that
> B is an alias for A, the load must be unwound and retried. (This doesn't increase the amount
> of state storage required, because the load also has to be unwound if the store B faults.)
>
> Let me enumerate the four cases for a load and a store to virtual aliases:
>

1. Store first, store hits in L1. When the load misses, it will go through to the L2, which will detect
   > the aliasing and force a write-back of the stored data before satisfying the load.
2. Load first, load hits
   > in L1. Very similar, but the write-back is simpler because L1 is clean.
3. Load first, load misses in L1.
   > Normally, the store is simply held in store buffers until all preceding loads complete, to avoid the
   > problem of having to unwind a store if the load faults. The load forces an L1 alias swap, and by the
   > time the store reaches L1, it misses and proceeds like the preceding case.
4. Store first, store misses
   > in L1. Unlike the preceding case, we really do want the load to proceed speculatively because performance
   > will suck if we don't. We satisfy the load from L1, but place a watchpoint on the corresponding L1 cache
   > line until earlier accesses are resolved. (This is probably implemented as a "load buffer" CAM of speculative
   > loads which is matched to incoming invalidates, as that it smaller than having space for multiple pending
   > loads in each cache line.) If the L1 cache line is evicted while it is being watched, the oldest speculative
   > load satisfied from that cache line is unwound and retried.

>
> The solution to the last case also suffices for the store buffer aliasing problem: the load remains
> speculative until all preceding accesses are resolved and the cache line is still valid in L1.
>
> It's also not unlike the "load linked/store conditional" mechanism.
>
> But this whole situation shows how nightmarish it is preventing Meltdown attacks in the presence of virtual
> aliasing. Because the store could take a long time (page table walking) to be fully resolved, and the load
> does not alias nine-nines of the time, we really can't afford to wait before proceeding speculatively.

It's not really bad. It is somewhat like spectre variant 4. This is no problem with such speculation within a privilege domain. Between domains like a system call then you need to ensure prior stores are executed at least to a point where speculative data would not be used.