Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Entropy and Related Concepts

1. Entropy
2. Shannon Entropy
3. Cross Entropy
4. K-L divergence

Tips

1. The entropy concept was first introduced for discrete distributions (called Shannon entropy), which is defined as

   $$H(X) = E[log(\frac{1}{f(x)})]$$
   where \(X\) stands for a discrete random variable (distribution) and \(f(x)\) is the probability density function of \(X\). Shannon Entropy is non-negative. It is zero if and only if the discrete distribution is degenerate (all mass concentrate on one point).

2. Shannon entropy is proven to be the lower bound of bits per symbol (\(log_2(x)\) is used instead of \(log(x)\)) to transfer identifiable information from a source to a destination through a communication channel without data loss.

3. Shanno entropy is equivalent to entropy in thermodynamics (an area of physics).

4. Entropy is a good metric to measure the magnitude of "information" in features/variables in machine learning. It can be used to filter out non-useful features/variables.

5. The entropy concept can be extended to continuous distributions. However, the entropy of a continuous distribution can be negative. As a matter of fact, the entropy of a continuous distribution has a range of \((-\infty, \infty)\). Taking the exponential distribution with the density function \(\frac{1}{\mu}e^{-\frac{x}{\mu}}\) as example, its entropy is \(log(\mu)+1\) which goes to \(\infty\) as \(\mu\) goes to \(\infty\) and it goes to \(-\infty\) as \(\mu\) goes to 0.

6. For the reason in bullet point 4, entropy is not a good measure for continuous distributions Cross-entropy and K-L divergence are more commonly used for both discrete and continuous distributions. The cross-entropy of a distribution q with respect to p is defined as

   $$H(p, q) = E_p[-log(q)]$$
   And the K-L divergence (also called relative entropy) is defined as
   $$D_{KL}(p, q) = E_p[log(\frac{1}{q}) - log(\frac{1}{p})] = H(p, q) - H(p)$$
   Notice that the K-L divergence is always non-negative.

7. In a multi-class classification problem, the following are equivalent.

   • minimizing the cross-entropy
   • minimizing the K-L divergence
   • maximizing the log likelihood of the corresponding multi-nomial distribution
   • minimizing the negative log likelihood (NLL) of the corresponding multi-nomial distribution

   The above conclusion suggests that the cross-entropy loss, K-L loss and the NLL loss are equivalent. However, be aware that PyTorch defines cross-entropy loss to be different from the NLL loss. The cross-entropy loss in PyTorch is defined on the raw output of a neural network layer while the NLL loss is defined on the output of a log softmax layer. This means that in PyTorch the cross-entropy loss is equivalent to log_softmax + nll_loss.

Misc

Fisher information explanation

likelihood based tests: LRT, wald, score

expected fisher,

observed fisher (sum, log, law of large number)

References

• A Gentle Introduction to Cross-Entropy for Machine Learning