Activation Functions in Neural Network

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

GELU¶

GELU is the best activation function currently (at least in NLP).

GELU(x) == x \Phi(x)

where $\Phi(x)$ is the cumulative distribution function of the standard normal distribution.

f(x) = x \dot \sigma(x)

, where $\sigma(x)$ is the sigmoid function.