David Kanter (dkanter.delete@this.realworldtech.com) on January 11, 2014 1:47 pm wrote:
> Nicolas Capens (nicolas.capens.delete@this.gmail.com) on January 11, 2014 12:24 am wrote:
> > David Kanter (dkanter.delete@this.realworldtech.com) on January 10, 2014 4:06 pm wrote:
> > > Nicolas Capens (nicolas.capens.delete@this.gmail.com) on January 10, 2014 2:22 pm wrote:
> > > > David Kanter (dkanter.delete@this.realworldtech.com) on January 9, 2014 6:42 pm wrote:
> > > > > > > I'll further add that I'm willing to wager that I'm correct about the 2 load
> > > > > > > pipelines, and that Nicolas is wrong about a virtually addressed L0 cache.
> > > > > >
> > > > > > What exactly do you mean by 2 load pipelines? If you mean that both vector units would be
> > > > > > able to use a memory operand in the same cycle, then yes, based on Eric's code that seems
> > > > > > to be a necessity. However due to the duplicate loading of the same memory this can be provided
> > > > > > by a dual-ported L0 cache, while the L1 cache itself only requires a single load port.
> > > > >
> > > > > I mean that the KNL core has two AGUs, both VPUs can execute
> > > > > load+op every cycle and that the L1D cache can fill
> > > > > two hits per cycle. I do not believe the cache is dual ported, as I believe it will be heavily banked.
> > > >
> > > > As far as I know that counts as a dual-ported cache.
> > >
> > > No it doesn't.
> >
> > It's what my professor called it: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.107.3602
> >
> > What did your professor call the thingies that determine the
> > maximum number of simultaneous accesses instead of ports?
>
> I didn't learn computer architecture from a professor.
>
> A cache and an SRAM are totally different entities, e.g., http://courses.engr.illinois.edu/ece512/spring2006/Papers/Itanium/isscc_2002_4s.pdf
> is a good explanation of the differences here (incidentally the distinction between SRAM and caches is something
> you seem to gloss over repeatedly throughout this thread).

With all due respect you don't have to lecture me on such basic matter. I did learn computer architecture from a professor, as well as digital design, embedded systems design, etc.

> The problem here is that CS people are incredibly imprecise and haphazard about terminology (in
> comparison to mathematics). SRAMs and RFs have ports. The definition there is fairly precise.

It helps to discern between hardware designers and software developers, instead of lumping them together as being "CS people", even though some are both. To a software developer, even a low-level compiler engineer, a cache is mostly a concept and has a broad meaning. You can even have a software cache. For low-level software developers it's important how many memory operands you can have per cycle, but it's of practically no importance how that's achieved in hardware. Bank conflicts should be rare enough for a software developer not to have to care about them. To a hardware designer a cache isn't just a concept it's a physical thing and if it's a banked design which can have a bank conflict that's of huge importance. Also, things evolve fast enough that the norm for a multi-ported cache implementation varies every few years. But it's still multi-ported by concept.

So I really don't think the problem is "CS people" being very imprecise about terminology. There's just different contexts, and it appears you're confusing a specific implementation for defining the meaning of certain terms. For what it's worth, Stanford professors consider multi-banking also as a method to implement a multi-ported cache: http://www.stanford.edu/class/ee282/handouts/lect.04.caches2.pdf

> I avoid using the same terminology on caches, or when I do, I try to make
> sure that I am precise (although I'm not successful all the time).
>
> What is a 'dual ported cache'?
>
> Is it a cache comprising a dual-ported tag and data array? That is the most common usage because
> most architects are thinking about circuits a fair bit. In fact, if you ask Intel's architects
> whether the caches are multi-ported, they will tell you 'no, they are multibanked'.
>
> Is it a cache that has two access 'ports'? Well maybe, but what are those 'ports' capable of? How do
> I know whether they are 2 R/W, R+W, 2R, or 2W? How do I know what the address constraints are? E.g.,
> is the underlying data array banked, while the tags are dual ported? What about the controller logic?
>
> There's a lot of confusing terminology here and it pays to be precise.

I'm afraid the confusion is all yours. Read ports are well defined and even if you lacked the terminology I think I've made myself perfectly clear from the start that the L1 cache may not require two reads per cycle to feed two vector units with (unique) data. The existence of an L0 cache with two read ports would potentially explain why the Intel compiler reads operands from memory again despite having the data in a register and despite L1 accesses consuming considerable power. And again, I'm not claiming KNL's to have such an L0 cache. I'm merely observing that in your article you're taking a leap by saying KNL must have two load pipelines (which you've clarified to mean two L1 load ports, not something like an L0 cache to service recently loaded data), which you've concluded solely from x86 being a load-op ISA. There are viable alternatives, of which I'm suggesting just one.

> > > > Anyway, a multi-ported multi-banked L1 cache is a reasonable possibility. I just don't see why it "must" be
> > > > the only possibility, especially with x86 being a load-op
> > > > architecture as the only explanation in >your article.
> > >
> > > > Given that 1 byte/FLOP would suffice and that the code generated
> > > > by ICC has a lot of duplicate >memory accesses,
> > > > it also seems reasonable to me that there's a single-ported L1 cache and a dual-ported L0 cache.
> > >
> > > First, 1B/FLOP isn't sufficient. Look at KNC, it can source 1x64B operand from memory
> > > and 2 from registers while performing 8 FMAs. That's 8B for 2 operations or 4B/FLOP.
> >
> > I was talking about single-precision FLOPS. And it's obvious that KNC had 2 bytes/SPFLOP since providing
> > any less would have required cutting that bus in half and sequencing things, which adds more complexity
> > overall.
>
> It turns to be the same thing, which is half an operand per FLOP.

Yes, but that's a consequence of the architecture. In contrast, Kepler has a SIMD array of 32 load units, which can fetch either 32 or 64-bit operands. So you have to take precision into account when making the comparison, and then you might as well use bytes/FLOP as the metric. GK110 has 0.33 bytes/SPFLOP and 2 bytes/DPFLOP. So with just one load port and two vector units KNL would beat it by having 1 byte/SPFLOP and 2 bytes/DPFLOP. There's no need to go overboard and double that, especially with an L0 cache to handle recently loaded data. KNC did have twice the L1 bandwidth but only because it only had one vector unit and providing a narrower load bus would have complicated things more than it would have saved. So again, that's not a reference. Kepler has low SP bandwidth/FLOP but that's more of a consequence of high SP FLOPs than low cache bandwidth, and works out fine for graphics workloads.

1 byte/SPFLOP, which amounts to one load operand for every two instruction, is the right ratio for the vast majority of workloads running on an architecture with ~32 registers. It's just an inherent property of the code, determined by the data locality in the algorithms. If you need more than 1 byte/SPFLOP, you should probably change something about your algorithms (like merging loops to avoid writing temporary results and having to read them back in the next one, or to average out the memory and arithmetic heavy code).

> >So KNC isn't the gold standard here. Doubling the number of execution units does not mean the
> > number of memory load ports have to be doubled. Kepler has one load port per six FMA units. I'm not saying
> > that's the best ratio, but it's a strong indication that 2 bytes/SPFLOP is completely unnecessary.
>
> Funny, but Intel isn't trying to build Kepler. They are building an HPC-specific part. I'm asserting
> the fact that KNC is the closest shipping approximation to what Intel believes is a good fit for HPC.

If KNC is so perfect then why didn't Intel win back the entire HPC market? I think it's a pipe dream that you can build anything that's head and shoulders above the competition. NVIDIA would have had no problems removing the texture units from Kepler and increasing the bytes/FLOP, if that's the recipe for success in the HPC market. Instead just like their previous architectures they decided to use the same chips for high-end graphics and HPC. Note that AMD doesn't wipe the floor with them either, despite GCN's higher bandwidth/FLOP.

The reality is that performance/Watt rules everything, and you don't win anything by providing more bandwidth than what the vast majority of the code uses because it burns power all the time.

> This should be obvious since that's what they invested millions of dollars in building.
> Notice how Intel specifically chose not to emulate Nvidia GPUs? Maybe it's because
> they think they have a different solution that they believe is a better fit?

Intel's only goal with MIC is to keep x86 relevant in the HPC market. They can't defeat NVIDIA through architecture alone. They even need their process advantage and software compatibility advantage just to have anything competitive and keep the world from switching to ARM and CUDA, or HSA for that matter. So this isn't offence, it's defense.

> > > Haswell also sources 64B/cycle while performing 8 FMAs.
> >
> > Yes, and it has a third and fourth arithmetic execution port, both of which can't have a
> > memory source operand if the other two already utilize the two L1 read ports.
>
> >So that's
> > a 2:4 ratio, while I'm suggesting a 1:2 ratio for KNL, augmented with an L0 cache so you
> > can actually have a 1:1 ratio to lower register pressure while staying power efficient.
>
> Notice how I was talking about bytes per FLOP? I'm sure you noticed but those other
> arithmetic units don't perform FLOATING POINT OPS. So the ratio still works fine.

Haswell has four arithmetic execution ports instead of two, despite 'only' two load ports, because they expect to have a purpose for them! Those two extra ports can also require a load operand. If it was really necessary to have two load ports just for the two floating-point execution ports, you couldn't execute many other instructions, let alone two! But Intel has explicitly mentioned that a fourth execution unit was added to offload the vector ports. As a consequence they also added a third AGU to sustain two loads per cycle, but nothing exceeding that 1:2 ratio.

So I think bytes/FLOP is only meaningful when every execution port is the same. Kepler for instance can do a floating-point or integer operation on any of its execution units, but obviously it's going to be rare for all of them to be doing floating-point exclusively. Also note that while FMA capabilities slash the bytes/FLOP figure in half, the number of FMA instructions in actual code is typically not that high (I recall a study saying 30% on average for floating-point heavy algorithms). So all I'm saying is that you shouldn't take the bytes/FLOP metric too literal. If one of Kepler's six SP execution units would be capable of integer operations only, that would increase their theoretical bytes/FLOP, but not result in a particularly better architecture.

> Moreover, even if you want to count them, it's still approximately the same.
> Haswell can perform 32 SPFLOP + 2 integer ops per clock. It can load 64B/clock.
> 64B / 34 ops = 1.88B/op. Notice how that's REALLY REALLY CLOSE to 2B/FLOP?

You can't count every SP operation individually and add that to the scalar operations. If an 8-bit integer instruction requires a memory source operand, it occupies an entire load port which could otherwise deliver 256-bit. In other words it's not worth 1 byte out of your 64 byte/clock bandwidth, but 32 bytes. You have to think in terms of instructions/cycle, regardless of whether it's scalar or vector instructions.

> > > So if you look at these designs, it's very clear that Intel believes that 4B/FLOP is the right
> > > amount of L1D bandwidth. Frankly, you can analyze compiler output all you want...but Intel
> > > invested millions in making the right decision and that's infinitely more convincing.
> >
> > So what do you think NVIDIA and AMD invested in making the right decision?
>
> NV and AMD are targeting different products with radically different workloads; they have to design products
> that are first and foremost good at DirectX. KNC and KNL don't run DirectX and therefore should be quite different.
> Also, Intel's resources are different than NV or AMD, so the solution will be different looking.

KNC is the remnant of Intel's failed attempt to create a discrete GPU. To have any success in the consumer market it had to be first and foremost a GPU. An x86 compatible GPGPU/HPC device would have just been a bonus. So what they considered the right decision for such a product back then shouldn't be very different from the decisions NVIDIA has to make to offer its architectures both as a consumer GPU and HPC device. However instead of something closer to 0.33 bytes/SPFLOP, Larrabee and KNC could do 2 bytes/SPFLOP. So that should really question your faith in their ability to make the right decision back then. If anything, the failure of Larrabee and lower bandwidth of GPUs and their relative success in the HPC market should be in indication they should do things differently next time. And again, with a single vector unit per core they didn't really have any other option, so stop considering it a right decision. There was no actual decision, and they ended up with something that failed as a GPU and barely made it as an HPC device.

Note also that Larrabee didn't primarily fail because of a bad architecture for executing programmable code, but because of a larger number of decisions such as the lack of several fixed-function units that made it significantly slower at running contemporary games. And by significant I mean enough for consumers to lose interest, which is actually a very small margin. The 'war' between NVIDIA and ATI has frequently been 'won' by only a 20% difference in bang for buck. Intel would have lost billions selling it in the price class for GPUs it could actually compete with. Any additional value from being more programmable would not have paid off for many years until Intel gained a dominant position and developers started to take notice. As an HPC device, only differences in the execution cores matter (and cache hierarchy), and they matter much less because more programmability itself has a lot of value.

My point is that just because NVIDIA and ATI got things right for a consumer graphics device, doesn't mean they got it wrong for HPC. Intel has the advantage of x86 compatibility, which helped compensate for any wrong decisions. If KNL has two vector units per core, thus offering them the option to save on expensive L1 bandwidth, there is no guarantee they'll go for 2 bytes/SPFLOP again just because x86 is a load-op ISA.

The stake are higher now for getting it right, because GPUs became a lot more programmable since the Larrabee days and Intel is now also competing with itself (people need a reason to upgrade from KNC, but will also soon have the option to get Xeon's with AVX-512). So KNL has to offer the highest possible performance/Watt for parallel workloads. With L1 accesses costing 40% more power, there's a lot to be gained from keeping the number of actual accesses low.

> > Looking at typical x86 assembly output teaches us a lot
> > of things. Not every instruction takes a memory source
> > operand. The ratio is rarely over 1:2 when the accesses are
> > unique, and having 32 registers helps a lot to ensure
> > that for AVX-512. However Eric's experiments show that the
> > Intel compiler for KNL isn't shy to use as many memory
> > operands as possible. And knowing how much an L1 access cost
> > in power, that doesn't make sense unless they have
> > 'something' to make it cheap for very locally reused data. I call that something an L0 cache.
>
> You are assuming that 'typical x86 assembly output' today is the same as what will ship in 2015. That's a
> bad assumption, unless you think that all of Intel's compiler people are going on vacation for a year.

This isn't a metric that varies wildly. An average MEM:ALU ratio of around 1:2 is very common across the board, because that's how many operations are performed on every input that isn't short-lived enough to be stored in a register. The Pentium had a 1:2 ratio, then they went to 1:3 with P6, up until Sandy Bridge which could squeeze in another load op when no store is happening, until finally Haswell restored the ratio to 1:2. GPUs also have a 1:2 ratio at best. So what would make KNL so different it needs a 1:1 ratio, giving it the ability to load data from memory on every single instruction, even though AVX-512F has 32 registers?

To me that only makes sense if the load ports are used in a novel way, such as with an L0 cache to efficiently reference recently loaded data again, therefore acting somewhat like a register cache. That is, you don't have to manually load data from memory into a register and reuse that, but you can keep referencing the memory and the hardware will figure out when it can fetch it from a tiny buffer instead of L1. This is the only way you can dramatically change the typical 1:2 ratio, which assumes optimizing for register reuse, without paying a hefty price.

> > > It won't be. L1 caches are fine for KNL and the code you're
> > > seeing is probably the result of an immature compiler.
> >
> > The Intel compiler has been around for a long time, and AVX-512 is a very minor variation on KNC's
> > 512-bit ISA extension. Also as someone pointed out elsewhere you need a somewhat representative
> > compiler to generate traces that can be used during the design of the new architecture. And lastly
> > this behavior isn't coming from some complicated experimental optimization. This is part of basic
> > register allocation and for decades the overall goal has been to minimize the number of memory operands.
> > To suddenly see that basic rule reversed cannot possibly be something Intel has overlooked. It's
> > a deliberate choice and we have to find the explanation in the micro-architecture.
>
> Or maybe it's just a compiler that still has a year worth of tuning to be done?

First of all this isn't a tiny tweak. There's a massive difference in how memory operands are used between the code generated for AVX2 and the code generated for AVX-512F. They need a somewhat representative compiler for designing the hardware, so they can't let this pass and make the change later. Secondly, it's straightforward for compiler engineers to switch back and forth between these two approaches. Not a year's worth or work.

> > > Caches might cost power, but the real problem is large scale coherency. The L1D is fine.
> >
> > The paper above says it takes 40% more power. That's a huge deal. Even if large scale coherency
> > is a bigger problem, you can't ignore this. 40% is a lot of opportunity for doing something a little
> > different when doubling the number of execution units, like, an L0 cache perhaps? Even if the L0
> > cache itself costs 10% of power per access, that's a huge saving over a second L1 access.
>
> Uh, the paper says that reading 512b from the L1D takes 40% more power than reading a register. First, that's
> pretty similar power consumption, and definitely not worth the complexity of an additional structure. Especially
> considering that there's also a store buffer running around - it would be nice if they modeled that.

You've got to be kidding. 40% is a huge difference. It definitely leaves room for a small structure like an L0 cache to help reduce that. Many substantial architectural changes offer a benefit that's only in the single digit percentages.

> I just don't see enough of a gain, especially since the structure you described
> adds overhead to register file reads, which is a terrible plan.

It doesn't add overhead to register file reads at all. It helps reduce register pressure.

> > > > > I'm willing to wager money that there is no L0 as you have described it.
> > > >
> > > > Gamblers wage a lot of money even though they know they have less than a 50% chance of winning
> > > > but choose to ignore it. So your money doesn't mean anything to me even though you say this
> > > > must be the only option for KNL. I won't wager anything because to me it's a coin on it's
> > > > side. I think a dual-ported L1 is not the only option. That doesn't mean I think the other
> > > > option is more likely. And neither does that doubt mean I think it's any less likely.
> > >
> > > What I hear is that you aren't very confident of the L0 being the right
> > > solution, whereas I'm highly confident that it is the wrong solution.
> >
> > There's a difference between it being the wrong solution and two L1 read ports being the right
> > solution. The L0 cache is just one way Intel might avoid a 40% increase in power consumption,
> > but the most likely one I've been able to come up with so far based on the compiler output.
> > That you think that output is "probably" the result of an immature compiler, which it clearly
> > isn't, is a much more interesting expression of doubt. So what's your other explanation?
>
> I'm confident there is no L0 cache. I never said that the L1D
> is dual ported, I said that it supports 2x64B reads/clock.

Then it needs two read ports. Even if it's implemented with multiple banks with a single read port each, it adds complexity to receive two addresses, select banks, and deal with conflicts, and you're also consuming twice the SRAM lookup power on every dual access in the same cycle. With a tiny L0 cache you could avoid all that and even reduce register file pressure.

> You're just going to have to accept that given the current set of facts today, there is no way you
> will convince me you are correct. I think the L0 cache is a terrible idea because it is a marginal
> gain, very brittle, and adds complexity to an area that is rife with critical paths already.

40% is not a marginal gain, there's nothing brittle about it since it can easily be kept coherent with L1, and it can fit in the LSU without affecting any critical paths. All your reasons for thinking it's a terrible idea are plain wrong.

> When the facts change, I might change my conclusion. Until then, I'm quite confident
> of where I stand. And I'm still open to that wager about the existence of the L0...

I'm not trying to convince you that I'm correct about the L0 cache. I'm trying to convince you that a dual-ported L1 is not a given due to x86 being a load-op ISA. Other x86 architectures, as well as GPUs, have a lower MEM:ALU ratio and they're doing fine. But a single L1 load port wouldn't explain the code Eric has posted. So I proposed the L0 cache as something which would explain all the given "facts" at once. Whether that's the solution Intel ended up implementing, I don't know. It's just one of the viable possibilities besides assuming it "must have" two L1 read ports.