Technology
The SELU Activation and Self-Normalizing Networks
The SELU (Scaled Exponential Linear Unit) function is defined as SELU(x) equal to lambda times ELU(x), with lambda near 1.0507 and alpha near 1.6733. Those two constants make activations converge on their own towards zero mean and unit variance layer after layer, building deep networks that normalize themselves without batch normalization.