Some power efficiency figures

By: Maynard Handley (, October 12, 2018 10:33 am
Room: Moderated Discussions
Simon Farnsworth ( on October 12, 2018 10:00 am wrote:
> AM ( on October 12, 2018 4:59 am wrote:
> > Simon Farnsworth ( on October 12, 2018 1:19 am wrote:
> > > anon ( on October 11, 2018 3:49 pm wrote:
> > > > AM ( on October 11, 2018 11:02 am wrote:
> > > > > Foo_ ( on October 11, 2018 10:22 am wrote:
> > > > > > AM ( on October 11, 2018 9:15 am wrote:
> > > > > > > And considering Cloudflare's well-known stance wrt ARM vs x86 (see e.g.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I suspect -- assuming Cloudflare folks are negotiation-smart -- they got these rejects not just for
> > > > > > > free, but for a hefty subzero price, part of their deal being this blog post mentioning they're upgrading
> > > > > > > their fleet to an off-roadmap/custom Intel CPU "offering considerable value in performance per watt".
> > > > > >
> > > > > > Why would Intel pay to advertise a CPU model that they don't want to sell?
> > > > >
> > > > > Not to advertise a "CPU model", but for Cloudflare staying with
> > > > > Intel -- their ARM intentions are well-known (link above).
> > > > >
> > > > > > > With Centriq, ThunderX2, and finally Ampere eMAG servers available (,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I guess the value of such PR project targeted at various business execs is well worth
> > > > > > > selling rejects at negative price to prevent someone size of Cloudflare switching to
> > > > > > > ARM.
> > > > > >
> > > > > > There are not many companies "size of Cloudflare" (in # of servers) around the world, I think.
> > > > > > And I doubt those companies make CPU architecture decisions by reading blog posts.
> > > > >
> > > > > Nothing more than sloppy wording on my part -- companies who consider, or are forced to consider,
> > > > > power efficiency and computer density are surely not limited to size of Cloudflare.
> > > >
> > > > They're all already looking at ARM, AMD, some even POWER.
> > > > They all have a strong interest to make non-Intel look
> > > > good on their workloads even if they don't switch, to negotiate with Intel. PR has nothing to do with it.
> > > >
> > > And, since we're looking at figures, the important figure of merit is whether the ARM server chip vendors
> > > are going to make enough money to sustain the business
> > > in the medium term. The fact that Qualcomm has scaled
> > > down its server division is a bad sign in this regard, as is the dropping out of other players. Is this
> > > chip really good enough to survive anything AMD and Intel are going to do in the next 5 years?
> > >
> > > Fundamentally, if you're moving to a new ISA to get competitive chips, you want a strong chance that there
> > > will be competitive products out there with the new ISA going forward. That means either a supplier that's
> > > showing an awareness that they could be waiting years for
> > > a return on investment and need to just keep going,
> > > or enough of a market that even if one supplier drops out, another will step in to reap the profits.
> > >
> > > Right now, x86 has both of those - Intel provides the first, and the market for x86 CPUs
> > > allows AMD to provide the second. ARM nearly had the second with Cavium and Qualcomm both
> > > in it, and until Qualcomm downscaled their server team, they had the first, too.
> >
> > These guys don't appear to have any issues with ISA or server vendor lock-in at all -- they
> > don't have any problem writing ARM asm just for the sake of studying what Centriq can offer,
> > getting x-fold gap in power efficiency vs Intel servers they owned among other findings.
> >
> Sure - playing around with a new ISA is cheap at that sort of scale. But committing to it (i.e.
> saying that everything must support ARM64 in future, and that you must ensure ARM64 performs
> well) is an ongoing cost, and you need a return on that investment. This generation of ARM64
> server chips is good enough that, if Qualcomm committed to keeping up with or exceeding Intel
> for each upcoming Intel generation, it would be worth Cloudflare's while moving over.
> What has gone wrong for Cloudflare's ARM movements so far is twofold:
> 1. Intel is willing to give Cloudflare a better deal to keep ARM out.
> 2. Qualcomm is not willing to commit resources to staying competitive - thus Cloudflare cannot
> be confident that, if Intel fix their 10nm and below mistakes, Qualcomm will continue to provide
> them with chips that equal or beat Intel for all the metrics Cloudflare cares about.
> That second is the big deal here. Cloudflare want to remain competitive into the future - it's a losing game
> if they switch to Centriq for the next year, then re-engineer completely onto future AMD or Intel chips.
> > And since I think it was you who set out to prove a while ago that ARM platform
> > was less power efficient than Intel's for [semi-]mobile devices we discussed, here
> > are a few links for you (as well as anyone else not following their work):
> >
> I didn't say that - I said that, given the devices I could find, the claim that the ARM
> platform was more power efficient than Intel was unproven (too much noise from the display
> and battery choices, in particular, to be confident that any perceived power efficiency
> difference was down to Intel versus ARM as opposed to other platform components).
> That does not hold for Centriq in Cloudflare's workloads as compared to Skylake, but there's no evidence
> that this is inherent to ARM, and not a result of different choices by Qualcomm and Intel, especially
> since, by Cloudflare's own statements, there are cases where Centriq would not fit at all but Intel
> can use the same core design to make a performant chip (low core count, high clocks).
> And I find it an interesting commentary on your zealotry that you assume that I, as an ARM advocate at work,
> would be biased towards Intel just because I asked for evidence and found counter-evidence to your claims…

We don't need to hypothesize about why Cloudflare did what. They explain it all in great detail here:

Some points that are (IMHO) either clear or implied:

- Cloudflare care about throughput more than latency. Once a core hits a particular minimum performance level, the main thing that matters is how many copies of that core (or its virtualized equivalent) can run; presumably because the work load consists of a lot of waiting on RAM in unpredicatable ways. This is important because this has ALWAYS been the claim as to where ARM can add value, that there are large customers who care primarily about throughput and don't need to pay the Intel tax to get an Intel-quality latency engine.
(And this isn't just talk. Whatever deal Intel and Cloudflare arranged this year, the bottom line is that Intel gave them SLOWER cores, but more of them, and Cloudflare took the deal.)

- Cloudflare care a great deal about compute density. "Maximizing compute density is the primary goal since rental colocation space and power are costly." You can project this beyond its obvious context ("Maximizing compute density is the primary goal [assuming the current context of minimum baseline performance, current power costs, current CPU costs]") and I fully expect someone will at some point to come up with a line like "well, if Cloudflare REALLY care only about compute density, why don't they fill their colo centers with Xeon Phi's or Platinum 8180M's? Obviously they are lying"...)

- Cloudflare operate on a large scale, and that has two consequences. One is that they operate with a standardized design, and they're not going to give up that design and all its benefits if they don't have to. That seems to be the real point behind the 1S or 2S business. Everything is structured around certain assumptions, and if you want to sell to them it's a lot easier to match those assumptions. Along the same lines, Cloudflare seems to have internalized the point (which is obvious, but surprisingly rare in the tech space) that there will always be a next year. ARM64 may be their future, it seems promising, but there will be a new round of purchases next year, and meanwhile this year's purchases have to go through no matter what.

- Finally (and CF are very careful how they say this, but I think the message is obvious, especially when you get to the comments) QC was, uh, not an optimal partner...
Essentially QC and Falkor showed the DESIRABILITY of an ARM64 throughput server, but not one from ARM. QC showed this in multiple ways. Clearly the 1S vs 2S nonsense showed that QC didn't seem to be interested in asking its most likely partners what they wanted, which didn't bode well for future collaboration and design improvement. I'm guessing along with that, there were a bunch of other issues where CF mentioned ways they want things done to QC, measured the reaction, and were not impressed. Meanwhile QC management seems to have been unclear, internally, for years, as to whether they actually want to do this or not (so they took the worst of all possible paths, wasting the development money, but not hitting a viable product; and they're STILL doing this --- insisting they haven't given up, while projecting such an air of chaos+lack of interest that no-one's going to buy from them at scale...)
Finally (and I don't know if this was an issue) QC's behavior re cellular chips may have given QC pause. The way QC has handled those sales, and the way it has lashed out in fury asserting patent thefts left and right (apparently, without any justification given by their recent court record), has probably left anyone who might have to buy from them at scale re-evaluating their options. Business is mostly an iterated game, and decent behavior and willingness to stick to standard business conventions matters more than one might expect.

So it looks like CF are hoping that Ampere will prove better partners, and that this is likely enough that it's worth maintaining and improving the ARM64 stack. Like I said, there is always a next year.
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Cloudflare jumps to custom Intelnobody in particular2018/10/10 08:55 AM
  Cloudflare jumps to custom Intelnone2018/10/10 09:25 AM
  Cloudflare jumps to custom Intelanon2018/10/10 09:39 AM
  Cloudflare jumps to custom IntelMaynard Handley2018/10/10 10:26 AM
    Cloudflare jumps to custom Intelnobody in particular2018/10/10 01:03 PM
      Cloudflare jumps to custom IntelWilco2018/10/10 01:25 PM
        Cloudflare jumps to custom Intelanon2018/10/10 02:07 PM
          Cloudflare jumps to custom Intelhobold2018/10/10 02:14 PM
          Cloudflare jumps to custom IntelWilco2018/10/10 02:58 PM
            Cloudflare jumps to custom Intelanon2018/10/10 08:56 PM
              Cloudflare jumps to custom IntelWilco2018/10/11 02:18 AM
                Cloudflare jumps to custom IntelMichael S2018/10/11 03:43 AM
                Cloudflare jumps to custom Intelanon2018/10/11 03:30 PM
          Cloudflare jumps to custom IntelBeastian2018/10/10 05:33 PM
            Cloudflare jumps to custom IntelDoug S2018/10/11 02:38 AM
          Qualcomm restructured server chip unitjuanrga2018/10/11 03:04 AM
            Qualcomm restructured server chip unitnone2018/10/11 04:38 AM
              Qualcomm restructured server chip unitMichael S2018/10/11 07:48 AM
                Qualcomm restructured server chip unitnone2018/10/11 09:15 AM
      Cloudflare jumps to custom Intelanon2018/10/10 01:29 PM
  Cloudflare jumps to custom IntelDoug S2018/10/10 12:10 PM
    Cloudflare jumps to custom Inteldmcq2018/10/10 12:27 PM
    Cloudflare jumps to custom Intelanon2018/10/10 03:07 PM
      Cloudflare jumps to custom IntelMichael S2018/10/11 04:06 AM
        Cloudflare jumps to custom Intelanon2018/10/11 07:02 AM
          Why can't two "computers" be on one board?Paul A. Clayton2018/10/11 07:49 PM
            Why can't two "computers" be on one board?rwessel2018/10/12 01:01 AM
              TwinBladesPaul A. Clayton2018/10/12 05:52 AM
            Why can't two "computers" be on one board?Doug S2018/10/12 02:01 AM
              Why can't two "computers" be on one board?Etienne2018/10/12 03:04 AM
              [O.T.] bladesMichael S2018/10/12 03:17 AM
                [O.T.] bladesAdrian2018/10/12 05:17 AM
              Why can't two "computers" be on one board?anon2018/10/12 04:04 AM
                Why can't two "computers" be on one board?rwessel2018/10/12 08:29 AM
                  Why can't two "computers" be on one board?anon2018/10/12 08:55 AM
              1S vs 2SWes Felter2018/10/12 02:30 PM
                1S vs 2SDoug S2018/10/12 03:35 PM
            Why can't two "computers" be on one board?anon2018/10/12 04:10 AM
            Why can't two "computers" be on one board?hobel2018/10/12 07:53 AM
      Cloudflare jumps to custom Intelanon.12018/10/11 10:16 AM
  Weird numbersjuanrga2018/10/11 03:11 AM
    Weird numbersanon2018/10/11 07:08 AM
      Weird numbersjuanrga2018/10/11 11:34 AM
        Weird numbersanon2018/10/11 12:24 PM
  Look at the figures, folksAM2018/10/11 09:15 AM
    Look at the figures, folksFoo_2018/10/11 10:22 AM
      Look at the figures, folksAM2018/10/11 11:02 AM
        Look at the figures, folksanon2018/10/11 03:49 PM
          Look at the figures, folksSimon Farnsworth2018/10/12 01:19 AM
            Some power efficiency figuresAM2018/10/12 04:59 AM
              Some power efficiency figuresSimon Farnsworth2018/10/12 10:00 AM
                Some power efficiency figuresMaynard Handley2018/10/12 10:33 AM
                  Some power efficiency figuresSimon Farnsworth2018/10/12 11:51 AM
                    Some power efficiency figuresanon2018/10/12 01:55 PM
                    Some power efficiency figuresMaynard Handley2018/10/12 03:37 PM
                      Some power efficiency figuresanon2018/10/12 04:04 PM
                        Some power efficiency figuresMaynard Handley2018/10/12 04:16 PM
                          Some power efficiency figuresanon2018/10/13 02:13 AM
                          Some power efficiency figuresanon2018/10/13 05:06 AM
                      Some power efficiency figuresSimon Farnsworth2018/10/13 04:52 AM
                        Some power efficiency figuresanon2018/10/13 05:17 AM
                Reread what you saidAM2018/10/14 09:14 AM
          Important clue for several postersAM2018/10/12 04:48 AM
            Important clue for several postersjuanrga2018/10/12 10:07 AM
              Important clue for several postersWilco2018/10/12 01:50 PM
                Thanks (NT)juanrga2018/10/13 04:17 AM
            Important clue for several postersWes Felter2018/10/12 02:42 PM
              Important clue for several postersAM2018/10/14 09:00 AM
                Cloudflare CPU has better perf/WDavid Kanter2018/10/15 08:39 AM
                  Cloudflare CPU has better perf/Wanon2018/10/15 09:10 AM
                    Cloudflare CPU has better perf/WDavid Kanter2018/10/15 11:09 AM
                      Cloudflare CPU has better perf/Wanon2018/10/15 12:52 PM
            Important clue for several postersanon2018/10/13 02:20 AM
              Important clue for several postersAM2018/10/14 08:58 AM
                Important clue for several postersanon2018/10/14 03:29 PM
                  Important clue for several postersAM2018/10/16 10:18 AM
                    Important clue for several postersanon2018/10/16 11:35 AM
                    Important clue for several postersanon2018/10/17 04:01 PM
    Look at the figures, folksDavid Kanter2018/10/13 06:08 AM
      Look at the figures, folksAM2018/10/14 08:50 AM
        Look at the figures, folksDavid Kanter2018/10/14 09:25 PM
          Look at the figures, folksMichael S2018/10/15 02:16 AM
            What?!AM2018/10/16 10:12 AM
          ARM servers in AzureMichael S2018/10/15 02:19 AM
            ARM servers in Azurejuanrga2018/10/15 05:41 AM
              ARM servers in AzureMichael S2018/10/15 05:46 AM
              ARM servers in AzureAnon2018/10/15 05:54 AM
                ARM servers in Azureanon2018/10/15 07:40 AM
          Look at the figures, folksAM2018/10/16 10:07 AM
        Look at the figures, folksMichael S2018/10/15 02:25 AM
        What figures?someone2018/10/15 05:35 AM
          What figures?AM2018/10/16 10:03 AM
            What figures?someone2018/10/17 08:19 AM
              What figures?wumpus2018/10/18 08:46 AM
      Look at the figures, folksJames2018/10/15 05:16 AM
        Look at the figures, folksAnne O. Nymous2018/10/15 05:31 AM
          Look at the figures, folksanonymou52018/10/15 05:47 AM
            Look at the figures, folksAnne O. Nymous2018/10/15 05:55 AM
        Watch the financial reportsAnon2018/10/15 06:00 AM
Reply to this Topic
Body: No Text
How do you spell avocado?