Infinity Fabric = PCIe

By: Adrian (, March 16, 2017 7:58 am
Room: Moderated Discussions
Michael S ( on March 16, 2017 4:32 am wrote:
> It depends on the number of signals in clock group. Intel demonstrated that with 5 data signal per group
> it's doable for much longer distances and much less controlled PCBs than one targeted by AMD.
> Hypertransport has 8+1 data signal per clock. That's a little harder than
> 5, but, then again, environment within MCM is very tightly controlled.

I agree that if you have few data signals, e.g. 4 per clock, that should be feasible.

Inside the MCM it is much easier, but we also have the links between 2 Naples sockets, where the PCB routing would be more difficult.

> As to "very large number of signals", I don't see how a total number of signals matters at all.

Well for 64 data signals, 16 other differential pairs for clocks would be needed. In a single socket system it would not make much sense for those 16 clocks to remain unused. It would have made more sense for those high-speed pins to be reconfigured to have 16 extra PCIe lanes.

You might still be right of course, but embedded clocks fit more closely the description given by AMD, where a single socket has up to 128 PCIe and two sockets have also up to 128 PCIe, because 64 are used by the inter-socket link.

It is also possible that separate clocks are used inside a MCM, for lower latency, and embedded clocks are used only between the 2 sockets, to make the PCB routing easier.


> I don't think so. 8 Gb/s is fast. Running faster will require more complex processing that will impact latency.
> If anything, I'd expect that inter-processor links would run slower than PCIe Gen3. I'd bet on 6.4 GT/s.

What you say is possible, but not very likely, because it would make the remote memory slower than the local memory. Even with the higher speeds mentioned by me, the remote memory would be slower in the unlikely situation when most cores from one package would attempt to access memory attached to the other package. But at least in more frequent cases the remote memory could have the same throughput as the local memory.

64 signals at 10.67 Gb/s would match 256 signals at 2.67 Gb/s, i.e. 4 memory channels, while each socket has 8 memory channels. So maximum memory throughput could be maintained when half of the memory is accessed locally and half remotely. With 6.4 Gb/s you will need to access not much more than a quarter of the memory remotely (and this ignoring overhead such as the coherence traffic).

We do not know the design goals of AMD, so maybe this limitation was considered to be OK and the speed is indeed lower as you believe.

< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
What is infinity fabric?slacker2017/03/12 06:41 PM
  What is infinity fabric?Jason Creighton2017/03/13 12:26 AM
    What is infinity fabric?Klimax2017/03/13 12:42 AM
      What is infinity fabric?Kevin G2017/03/13 04:39 AM
  What is infinity?Anon2017/03/13 04:44 AM
  What is infinity fabric?Groo2017/03/13 07:33 AM
    What is infinity fabric?Ireland2017/03/13 09:12 AM
      Another way to put same questionIreland2017/03/13 09:16 AM
      What is infinity fabric?Groo2017/03/13 12:12 PM
        What is infinity fabric?Ireland2017/03/13 02:07 PM
          What is infinity fabric?Ireland2017/03/13 02:32 PM
            What is infinity fabric?Seni2017/03/14 12:38 PM
          What is infinity fabric?Brendan2017/03/14 05:28 AM
            What is infinity fabric?Ireland2017/03/14 06:09 AM
            What is infinity fabric?Groo2017/03/14 07:35 AM
              What is infinity fabric?Ireland2017/03/14 08:35 AM
    Infinity Fabric = PCIeslacker2017/03/15 02:59 PM
      Infinity Fabric = PCIeMichael S2017/03/15 04:19 PM
        Infinity Fabric = PCIeIreland2017/03/15 04:49 PM
        Infinity Fabric = PCIeRicardo B2017/03/15 10:25 PM
        Infinity Fabric = PCIeslacker2017/03/15 10:45 PM
          Infinity Fabric = PCIeRicardo B2017/03/15 11:02 PM
            Infinity Fabric = PCIeAaron Spink2017/03/16 01:08 AM
              Infinity Fabric = PCIeRicardo B2017/03/16 05:30 AM
                Infinity Fabric = PCIeAaron Spink2017/03/16 10:58 AM
        Infinity Fabric = PCIeGabriele Svelto2017/03/16 03:48 AM
          Latency vs. bandwidthDavid Kanter2017/03/16 08:04 AM
      Infinity Fabric = PCIeGroo2017/03/15 05:35 PM
        Infinity Fabric = PCIeslacker2017/03/15 11:11 PM
          Infinity Fabric = PCIeAdrian2017/03/16 02:51 AM
            Infinity Fabric = PCIeMichael S2017/03/16 04:32 AM
              Infinity Fabric = PCIeAdrian2017/03/16 07:58 AM
                Infinity Fabric = PCIeRicardo B2017/03/16 09:16 AM
                  Infinity Fabric = PCIeMichael S2017/03/16 09:54 AM
                    Infinity Fabric = PCIeRicardo B2017/03/16 10:07 AM
                      Infinity Fabric = PCIeMichael S2017/03/16 02:33 PM
                        Infinity Fabric = PCIeGabriele Svelto2017/03/17 03:35 AM
                          Infinity Fabric = PCIeAaron Spink2017/03/17 10:25 AM
                            Thanks, your posts on this thread have been extremely informative (NT)Gabriele Svelto2017/03/17 01:08 PM
                            Infinity Fabric = PCIeRicardo B2017/03/17 06:08 PM
                              Infinity Fabric = PCIeAaron Spink2017/03/17 06:27 PM
                              Quickpath papers - circuits and implementationslacker2017/03/18 06:50 PM
                                Quickpath papers - circuits and implementationRicardo B2017/03/19 08:07 AM
                    Infinity Fabric = PCIeAaron Spink2017/03/16 11:11 AM
                      Infinity Fabric = PCIeMichael S2017/03/16 02:46 PM
                        Infinity Fabric = PCIeRicardo B2017/03/16 03:07 PM
                          Infinity Fabric = PCIeMichael S2017/03/16 05:13 PM
                            Infinity Fabric = PCIeAaron Spink2017/03/16 11:04 PM
                            Infinity Fabric = PCIeRicardo B2017/03/16 11:30 PM
                              Infinity Fabric = PCIeMichael S2017/03/17 07:24 AM
                                Infinity Fabric = PCIeRicardo B2017/03/17 06:07 PM
              Infinity Fabric = PCIeDavid Kanter2017/03/16 08:36 AM
                Infinity Fabric = PCIeMichael S2017/03/16 08:50 AM
                  Infinity Fabric = PCIeAaron Spink2017/03/16 11:24 AM
                    Infinity Fabric = PCIeMichael S2017/03/16 02:45 PM
                      Infinity Fabric = PCIeAaron Spink2017/03/16 11:21 PM
Reply to this Topic
Body: No Text
How do you spell green?