Entropy

If p is a discrete probability measure, then the Shannon entropy of p is H(p) =  − ∑_xp({x})log p({x}). I’ve never had any intuitive feeling for Shannon entropy until I noticed the well-known fact that H(p) is the expected value of the logarithmic inaccuracy score of p by the lights of p. Since I’ve spent a long time thinking about inaccuracy scores, I now get some intuitions about entropy for free.

Entropy is a measure of the randomness of p. But now I am thinking that there are other measures: For any strictly proper inaccuracy scoring rule s, we can take E_ps(p) to be some sort of a measure of the randomness of p. These won’t have the nice connections with information theory, though.