Nicolas Capens (nicolas.capens.delete@this.gmail.com) on January 11, 2014 12:24 am wrote:
> David Kanter (dkanter.delete@this.realworldtech.com) on January 10, 2014 4:06 pm wrote:
> > Nicolas Capens (nicolas.capens.delete@this.gmail.com) on January 10, 2014 2:22 pm wrote:
> > > David Kanter (dkanter.delete@this.realworldtech.com) on January 9, 2014 6:42 pm wrote:
> > > > > > I'll further add that I'm willing to wager that I'm correct about the 2 load
> > > > > > pipelines, and that Nicolas is wrong about a virtually addressed L0 cache.
> > > > >
> > > > > What exactly do you mean by 2 load pipelines? If you mean that both vector units would be
> > > > > able to use a memory operand in the same cycle, then yes, based on Eric's code that seems
> > > > > to be a necessity. However due to the duplicate loading of the same memory this can be provided
> > > > > by a dual-ported L0 cache, while the L1 cache itself only requires a single load port.
> > > >
> > > > I mean that the KNL core has two AGUs, both VPUs can execute
> > > > load+op every cycle and that the L1D cache can fill
> > > > two hits per cycle. I do not believe the cache is dual ported, as I believe it will be heavily banked.
> > >
> > > As far as I know that counts as a dual-ported cache.
> >
> > No it doesn't.
>
> It's what my professor called it: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.107.3602
>
> What did your professor call the thingies that determine the
> maximum number of simultaneous accesses instead of ports?

I didn't learn computer architecture from a professor.

A cache and an SRAM are totally different entities, e.g., http://courses.engr.illinois.edu/ece512/spring2006/Papers/Itanium/isscc_2002_4s.pdf is a good explanation of the differences here (incidentally the distinction between SRAM and caches is something you seem to gloss over repeatedly throughout this thread).

The problem here is that CS people are incredibly imprecise and haphazard about terminology (in comparison to mathematics). SRAMs and RFs have ports. The definition there is fairly precise.

I avoid using the same terminology on caches, or when I do, I try to make sure that I am precise (although I'm not successful all the time).

What is a 'dual ported cache'?

Is it a cache comprising a dual-ported tag and data array? That is the most common usage because most architects are thinking about circuits a fair bit. In fact, if you ask Intel's architects whether the caches are multi-ported, they will tell you 'no, they are multibanked'.

Is it a cache that has two access 'ports'? Well maybe, but what are those 'ports' capable of? How do I know whether they are 2 R/W, R+W, 2R, or 2W? How do I know what the address constraints are? E.g., is the underlying data array banked, while the tags are dual ported? What about the controller logic?

There's a lot of confusing terminology here and it pays to be precise.

> > > Anyway, a multi-ported multi-banked L1 cache is a reasonable possibility. I just don't see why it "must" be
> > > the only possibility, especially with x86 being a load-op
> > > architecture as the only explanation in >your article.
> >
> > > Given that 1 byte/FLOP would suffice and that the code generated
> > > by ICC has a lot of duplicate >memory accesses,
> > > it also seems reasonable to me that there's a single-ported L1 cache and a dual-ported L0 cache.
> >
> > First, 1B/FLOP isn't sufficient. Look at KNC, it can source 1x64B operand from memory
> > and 2 from registers while performing 8 FMAs. That's 8B for 2 operations or 4B/FLOP.
>
> I was talking about single-precision FLOPS. And it's obvious that KNC had 2 bytes/SPFLOP since providing
> any less would have required cutting that bus in half and sequencing things, which adds more complexity
> overall.

It turns to be the same thing, which is half an operand per FLOP.

>So KNC isn't the gold standard here. Doubling the number of execution units does not mean the
> number of memory load ports have to be doubled. Kepler has one load port per six FMA units. I'm not saying
> that's the best ratio, but it's a strong indication that 2 bytes/SPFLOP is completely unnecessary.

Funny, but Intel isn't trying to build Kepler. They are building an HPC-specific part. I'm asserting the fact that KNC is the closest shipping approximation to what Intel believes is a good fit for HPC.

This should be obvious since that's what they invested millions of dollars in building. Notice how Intel specifically chose not to emulate Nvidia GPUs? Maybe it's because they think they have a different solution that they believe is a better fit?

> > Haswell also sources 64B/cycle while performing 8 FMAs.
>
> Yes, and it has a third and fourth arithmetic execution port, both of which can't have a
> memory source operand if the other two already utilize the two L1 read ports.

>So that's
> a 2:4 ratio, while I'm suggesting a 1:2 ratio for KNL, augmented with an L0 cache so you
> can actually have a 1:1 ratio to lower register pressure while staying power efficient.

Notice how I was talking about bytes per FLOP? I'm sure you noticed but those other arithmetic units don't perform FLOATING POINT OPS. So the ratio still works fine.

Moreover, even if you want to count them, it's still approximately the same. Haswell can perform 32 SPFLOP + 2 integer ops per clock. It can load 64B/clock. 64B / 34 ops = 1.88B/op. Notice how that's REALLY REALLY CLOSE to 2B/FLOP?

> > So if you look at these designs, it's very clear that Intel believes that 4B/FLOP is the right
> > amount of L1D bandwidth. Frankly, you can analyze compiler output all you want...but Intel
> > invested millions in making the right decision and that's infinitely more convincing.
>
> So what do you think NVIDIA and AMD invested in making the right decision?

NV and AMD are targeting different products with radically different workloads; they have to design products that are first and foremost good at DirectX. KNC and KNL don't run DirectX and therefore should be quite different. Also, Intel's resources are different than NV or AMD, so the solution will be different looking.

> Looking at typical x86 assembly output teaches us a lot of things. Not every instruction takes a memory source
> operand. The ratio is rarely over 1:2 when the accesses are unique, and having 32 registers helps a lot to ensure
> that for AVX-512. However Eric's experiments show that the Intel compiler for KNL isn't shy to use as many memory
> operands as possible. And knowing how much an L1 access cost in power, that doesn't make sense unless they have
> 'something' to make it cheap for very locally reused data. I call that something an L0 cache.

You are assuming that 'typical x86 assembly output' today is the same as what will ship in 2015. That's a bad assumption, unless you think that all of Intel's compiler people are going on vacation for a year.

> > It won't be. L1 caches are fine for KNL and the code you're
> > seeing is probably the result of an immature compiler.
>
> The Intel compiler has been around for a long time, and AVX-512 is a very minor variation on KNC's
> 512-bit ISA extension. Also as someone pointed out elsewhere you need a somewhat representative
> compiler to generate traces that can be used during the design of the new architecture. And lastly
> this behavior isn't coming from some complicated experimental optimization. This is part of basic
> register allocation and for decades the overall goal has been to minimize the number of memory operands.
> To suddenly see that basic rule reversed cannot possibly be something Intel has overlooked. It's
> a deliberate choice and we have to find the explanation in the micro-architecture.

Or maybe it's just a compiler that still has a year worth of tuning to be done?

> > Caches might cost power, but the real problem is large scale coherency. The L1D is fine.
>
> The paper above says it takes 40% more power. That's a huge deal. Even if large scale coherency
> is a bigger problem, you can't ignore this. 40% is a lot of opportunity for doing something a little
> different when doubling the number of execution units, like, an L0 cache perhaps? Even if the L0
> cache itself costs 10% of power per access, that's a huge saving over a second L1 access.

Uh, the paper says that reading 512b from the L1D takes 40% more power than reading a register. First, that's pretty similar power consumption, and definitely not worth the complexity of an additional structure. Especially considering that there's also a store buffer running around - it would be nice if they modeled that.

I just don't see enough of a gain, especially since the structure you described adds overhead to register file reads, which is a terrible plan.

> > > > I'm willing to wager money that there is no L0 as you have described it.
> > >
> > > Gamblers wage a lot of money even though they know they have less than a 50% chance of winning
> > > but choose to ignore it. So your money doesn't mean anything to me even though you say this
> > > must be the only option for KNL. I won't wager anything because to me it's a coin on it's
> > > side. I think a dual-ported L1 is not the only option. That doesn't mean I think the other
> > > option is more likely. And neither does that doubt mean I think it's any less likely.
> >
> > What I hear is that you aren't very confident of the L0 being the right
> > solution, whereas I'm highly confident that it is the wrong solution.
>
> There's a difference between it being the wrong solution and two L1 read ports being the right
> solution. The L0 cache is just one way Intel might avoid a 40% increase in power consumption,
> but the most likely one I've been able to come up with so far based on the compiler output.
> That you think that output is "probably" the result of an immature compiler, which it clearly
> isn't, is a much more interesting expression of doubt. So what's your other explanation?

I'm confident there is no L0 cache. I never said that the L1D is dual ported, I said that it supports 2x64B reads/clock.

You're just going to have to accept that given the current set of facts today, there is no way you will convince me you are correct. I think the L0 cache is a terrible idea because it is a marginal gain, very brittle, and adds complexity to an area that is rife with critical paths already.

When the facts change, I might change my conclusion. Until then, I'm quite confident of where I stand. And I'm still open to that wager about the existence of the L0...

David