David Kanter (dkanter.delete@this.realworldtech.com) on January 10, 2014 4:06 pm wrote:
> Nicolas Capens (nicolas.capens.delete@this.gmail.com) on January 10, 2014 2:22 pm wrote:
> > David Kanter (dkanter.delete@this.realworldtech.com) on January 9, 2014 6:42 pm wrote:
> > > > > I'll further add that I'm willing to wager that I'm correct about the 2 load
> > > > > pipelines, and that Nicolas is wrong about a virtually addressed L0 cache.
> > > >
> > > > What exactly do you mean by 2 load pipelines? If you mean that both vector units would be
> > > > able to use a memory operand in the same cycle, then yes, based on Eric's code that seems
> > > > to be a necessity. However due to the duplicate loading of the same memory this can be provided
> > > > by a dual-ported L0 cache, while the L1 cache itself only requires a single load port.
> > >
> > > I mean that the KNL core has two AGUs, both VPUs can execute
> > > load+op every cycle and that the L1D cache can fill
> > > two hits per cycle. I do not believe the cache is dual ported, as I believe it will be heavily banked.
> >
> > As far as I know that counts as a dual-ported cache.
>
> No it doesn't.

It's what my professor called it: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.107.3602

What did your professor call the thingies that determine the maximum number of simultaneous accesses instead of ports?

> > Anyway, a multi-ported multi-banked L1 cache is a reasonable possibility. I just don't see why it "must" be
> > the only possibility, especially with x86 being a load-op
> > architecture as the only explanation in >your article.
>
> > Given that 1 byte/FLOP would suffice and that the code generated
> > by ICC has a lot of duplicate >memory accesses,
> > it also seems reasonable to me that there's a single-ported L1 cache and a dual-ported L0 cache.
>
> First, 1B/FLOP isn't sufficient. Look at KNC, it can source 1x64B operand from memory
> and 2 from registers while performing 8 FMAs. That's 8B for 2 operations or 4B/FLOP.

I was talking about single-precision FLOPS. And it's obvious that KNC had 2 bytes/SPFLOP since providing any less would have required cutting that bus in half and sequencing things, which adds more complexity overall. So KNC isn't the gold standard here. Doubling the number of execution units does not mean the number of memory load ports have to be doubled. Kepler has one load port per six FMA units. I'm not saying that's the best ratio, but it's a strong indication that 2 bytes/SPFLOP is completely unnecessary.

> Haswell also sources 64B/cycle while performing 8 FMAs.

Yes, and it has a third and fourth arithmetic execution port, both of which can't have a memory source operand if the other two already utilize the two L1 read ports. So that's a 2:4 ratio, while I'm suggesting a 1:2 ratio for KNL, augmented with an L0 cache so you can actually have a 1:1 ratio to lower register pressure while staying power efficient.

> So if you look at these designs, it's very clear that Intel believes that 4B/FLOP is the right
> amount of L1D bandwidth. Frankly, you can analyze compiler output all you want...but Intel
> invested millions in making the right decision and that's infinitely more convincing.

So what do you think NVIDIA and AMD invested in making the right decision?

Looking at typical x86 assembly output teaches us a lot of things. Not every instruction takes a memory source operand. The ratio is rarely over 1:2 when the accesses are unique, and having 32 registers helps a lot to ensure that for AVX-512. However Eric's experiments show that the Intel compiler for KNL isn't shy to use as many memory operands as possible. And knowing how much an L1 access cost in power, that doesn't make sense unless they have 'something' to make it cheap for very locally reused data. I call that something an L0 cache.

> > L1 accesses cost a considerable amount of power (http://www.eecs.harvard.edu/~shao/papers/shao2013-islped.pdf),
> >
> > and if Intel wants to win the exascale race then they have to consider anything that might
> > offer an improvement in performance/Watt. An efficient L0 cache and tuning the compiler
> > to maximize its use might be part of such innovations.
>
> It won't be. L1 caches are fine for KNL and the code you're
> seeing is probably the result of an immature compiler.

The Intel compiler has been around for a long time, and AVX-512 is a very minor variation on KNC's 512-bit ISA extension. Also as someone pointed out elsewhere you need a somewhat representative compiler to generate traces that can be used during the design of the new architecture. And lastly this behavior isn't coming from some complicated experimental optimization. This is part of basic register allocation and for decades the overall goal has been to minimize the number of memory operands. To suddenly see that basic rule reversed cannot possibly be something Intel has overlooked. It's a deliberate choice and we have to find the explanation in the micro-architecture.

> Caches might cost power, but the real problem is large scale coherency. The L1D is fine.

The paper above says it takes 40% more power. That's a huge deal. Even if large scale coherency is a bigger problem, you can't ignore this. 40% is a lot of opportunity for doing something a little different when doubling the number of execution units, like, an L0 cache perhaps? Even if the L0 cache itself costs 10% of power per access, that's a huge saving over a second L1 access.

> > > I'm willing to wager money that there is no L0 as you have described it.
> >
> > Gamblers wage a lot of money even though they know they have less than a 50% chance of winning
> > but choose to ignore it. So your money doesn't mean anything to me even though you say this
> > must be the only option for KNL. I won't wager anything because to me it's a coin on it's
> > side. I think a dual-ported L1 is not the only option. That doesn't mean I think the other
> > option is more likely. And neither does that doubt mean I think it's any less likely.
>
> What I hear is that you aren't very confident of the L0 being the right
> solution, whereas I'm highly confident that it is the wrong solution.

There's a difference between it being the wrong solution and two L1 read ports being the right solution. The L0 cache is just one way Intel might avoid a 40% increase in power consumption, but the most likely one I've been able to come up with so far based on the compiler output. That you think that output is "probably" the result of an immature compiler, which it clearly isn't, is a much more interesting expression of doubt. So what's your other explanation?