The indescribable stupidity of bfloat16

By: Hans de Vries (nospam.delete@this.thank.you.so), February 18, 2020 5:35 am
Room: Moderated Discussions
The indescribable stupidity of bfloat16. An AI data format which shows a total lack of Intelligence.

The input for AI applications is sensor data and bfloat16 is specifically meant for visual sensor data. It is supposed to replace fp16, uint8 and int16 with something superior…

It is an ad-hoc standard not based on any scientific need but rather on ignorant programmer’s geekiness (easy to translate to from FP32) and cheap marketing considerations (bfloat16 MAC’s are so cheap that we can inflate our chip’s TOPS number)

The human eye resolves more than bfloats16’s 7 bit mantissa and therefore AI applications using bfloat16 can in principle be blind for visual information that we as humans can see.

Adding many numbers (convolution) with low precision can give very large errors depending on the order of addition. (small numbers first is okay, big numbers first is not okay)

Instead of improving the precision, bfloat16 instead gives us a totally absurd gigantic dynamic range due to the extreme overkill of the exponent (2^256 = ~ 10^77)

From the point of a vision sensor, the total dynamic range bfloat16 is enough to cover a camera input which ranges from detecting a single photon per photo cell to detecting all of the photons of the Universe together on a single photo cell per frame, give or take a few orders.

1) Imagine concentrating all the light of the sun's entire surface on a single pixel.
2) Multiply the intensity by ~100,000,000 to account for the light of all stars in our galaxy.
3) Multiply the intensity again by a factor of ~1,000,000,000 to account for all the galaxies.

We’re not there yet oh no ….

4) Multiply the intensity again by 60 (or more) for the number of camera frames per second.
5) Multiply the intensity again by ~30,000,000 for the number of seconds per year.
6) Multiply the intensity again by ~10,000,000,000 years that stars have been radiating.

That’s the dynamic range we get by throwing away the precision in bfloat16 ….

Think about this the next time you hear the usual marketing bullshit about how important bfloat16 (instead of just fp16) is for Artificial Intelligence.
 Next Post in Thread >
TopicPosted ByDate
The indescribable stupidity of bfloat16Hans de Vries2020/02/18 05:35 AM
  The indescribable stupidity of bfloat16Michael S2020/02/18 06:48 AM
  The indescribable stupidity of bfloat16Andrew Clough2020/02/18 07:17 AM
  Not so simpleMark Roulo2020/02/18 08:11 AM
    Not so simpleChris Dewan2020/02/18 08:49 AM
  The indescribable stupidity of bfloat16NoSpammer2020/02/18 08:48 AM
    The indescribable stupidity of bfloat16dmcq2020/02/18 11:32 AM
      The indescribable stupidity of bfloat16Hans de Vries2020/02/18 01:19 PM
  The indescribable stupidity of bfloat16Paul2020/02/18 12:19 PM
    This is mostly about bandwidthMark Roulo2020/02/19 09:00 AM
  The indescribable stupidity of bfloat16Not really2020/02/18 07:39 PM
    The single bfloat16 compute function VDPBF16PSHans de Vries2020/02/18 11:04 PM
      The single bfloat16 compute function VDPBF16PSJan Vlietinck2020/02/18 11:51 PM
      "neural" and "intelligence"hobold2020/02/19 12:17 PM
  bfloat and graphicsDavid Kanter2020/02/20 10:57 PM
Reply to this Topic
Name:
Email:
Topic:
Body: No Text
How do you spell purple?