Nvidia H100 Tensor Core GPU

By: Marcus (m.delete@this.bitsnbites.eu), March 24, 2022 3:03 am
Room: Moderated Discussions
dmcq (dmcq.delete@this.fano.co.uk) on March 24, 2022 1:40 am wrote:
> Marcus (m.delete@this.bitsnbites.eu) on March 23, 2022 11:23 pm wrote:
> > Hopper (hopper.delete@this.hopper.com) on March 22, 2022 8:48 am wrote:
> > > https://www.nvidia.com/en-us/data-center/h100/
> >
> > I see that they included an FP8 format, and the E4M3 variant uses the exact same binary format as
> > the float8 format in the MRISC32 ISA. At the time it was introduced in MRISC32 I was unsure about
> > what exponent/mantissa configuration to use, but E4M3 seemed like the most sane configuration.
> >
> > ...though they seem to support a nice higher-precision accumulator
> > for matrix multiplication, which makes perfect sense.
>
> I'd be interested if anyone had started using stochastic rounding
> yet, I'd have thought it should be the preferred mode in learning.

I think the main problem with that is reproducibility. In many situations you want to be able to guarantee that two consecutive training passes produce the same result. Otherwise, I agree.
< Previous Post in ThreadNext Post in Thread >
TopicPosted ByDate
Nvidia H100 Tensor Core GPUHopper2022/03/22 08:48 AM
  Nvidia H100 Tensor Core GPUMarcus2022/03/23 11:23 PM
    Nvidia H100 Tensor Core GPUdmcq2022/03/24 01:40 AM
      Nvidia H100 Tensor Core GPUMarcus2022/03/24 03:03 AM
        Pseudo-random number generator would provide reproducible stochastic rounding (NT)Paul A. Clayton2022/03/24 11:43 AM
          Pseudo-random number generator would provide reproducible stochastic roundingrwessel2022/03/24 01:28 PM
            Pseudo-random number generator would provide reproducible stochastic roundingAdrian2022/03/24 02:34 PM
              Pseudo-random number generator would provide reproducible stochastic roundingMarcus2022/03/25 12:57 AM
                Pseudo-random number generator would provide reproducible stochastic roundingAdrian2022/03/25 04:16 AM
                  Pseudo-random number generator would provide reproducible stochastic roundingMarcus2022/03/25 05:48 AM
                    Pseudo-random number generator would provide reproducible stochastic roundingAdrian2022/03/25 09:37 AM
                      stateless PRNGshobold2022/03/25 02:34 PM
                        stateless PRNGsJörn Engel2022/03/25 09:30 PM
                          stateless PRNGshobold2022/03/26 10:32 AM
                            stateless PRNGsJörn Engel2022/03/26 02:14 PM
                              stateless PRNGshobold2022/03/27 02:11 AM
                      Pseudo-random number generator would provide reproducible stochastic roundingblaine2022/03/26 01:09 PM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell tangerine? 🍊