Developer Guide

Contents

Batch Normalization Forward Layer

The forward batch normalization layer [ Ioffe2015 ] normalizes
x
i
1
...
i
p
from the input
X
R
n
1
x
n
2
x ... x
n
p
for the dimension
k
∈ {1, ...
p
} and then scales and shifts the result of the normalization using the provided weights and biases as follows:
where the following characteristics are computed for the input
X
:
  • means
  • standard deviation
    with variance
  • a constant ε to improve the numerical stability
The weights and biases are learned, as well as the rest model parameters.

Problem Statement

Given a
p
-dimensional tensor
X
R
n
1
x
n
2
x ... x
n
p
, the problem is to compute the
p
-dimensional tensor
Y
R
n
1
x
n
2
x ... x
n
p
:
where:
  • mean
  • variance
  • standard deviation
  • weights
  • biases
At the model training stage, along with the normalizing, the layer computes the population mean and variance using the exponential moving average with smoothing factor α ∈ [0,1] applied to the mini-batch means and variances.

Product and Performance Information

1

Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804