bfloat16 - Hardware Numerics Definition

Download
  • File: bf16-hardware-numerics-definition-white-paper.pdf
  • Size:229.61 KB

Details

Intel® Deep Learning Boost (Intel® DL Boost) uses bfloat16 format (BF16). This document describes the bfloat16 floating-point format.

BF16 has several advantages over FP16:

  • It can be seen as a short version of FP32, skipping the least significant 16 bits of mantissa.
  • There is no need to support denormals; FP32, and therefore also BF16, offer more than enough range for deep learning training tasks.
  • FP32 accumulation after the multiply is essential to achieve sufficient numerical behavior on an application level.
  • Hardware exception handling is not needed as this is a performance optimization; industry is designing algorithms around checking inf/NaN.

For more complete information about compiler optimizations, see our Optimization Notice.