Information theory
Entropy Differential entropy Conditional entropy Joint entropy Mutual information Conditional mutual information Entropy rate
Asymptotic equipartition property Rate–distortion theory
Shannon's source coding theorem Noisy-channel coding theorem
v t e

Mutual information measures how much more is known about one random value when given another. For example, knowing the temperature of a random day of the year will not reveal what month it is, but it will give some hint. In the same way, knowing what month it is will not reveal the exact temperature, but will make certain temperatures more or less likely. These hints or changes in likelihood are explained and measured with mutual information.

To calculate mutual information, the probability (chance) of all possible events, and the probability of them happening at the same time, is needed. For example, to measure mutual information between month and temperature we would need to know how many days in the year are 10 degrees Celsius, how many days out of the year are March and finally how many days are 10 degrees Celsius in March.

The formula requires the summation, or adding up, of many terms or numbers. Every possible outcome has its own term. From the above calculation of mutual information between month and temperature, we will use the following variables:

• p(x,y) = probability of it being x degrees Celsius in month y
• t(x) = probability of it being x degrees Celsius (on any day of the year)
• m(y) = probability of it being month y

This means m(3) equals the probability of a randomly selected day being in March. This is 31/365, or about 0.085, since 31 out of 365 days in the year are in March.

One term would be as follows:

${\displaystyle p(x,y)\log {\left({\frac {p(x,y)}{t(x)m(y)))\right)}\,\!}$

In this formula, "log" means logarithm.

Adding all possible terms together gives the value for mutual information.

The larger the mutual information, the more you would learn about one random value (e.g. month) when told another (e.g. temperature).

• If mutual information is zero, you cannot determine anything about one value when given another.
  • For example, knowing if you got heads or tails in a coin flip last time will not tell you anything about what will happen next time.

• If mutual information is small, there may still be no mutual information. Sometimes random events seem to have a pattern in the short term, but overall there is no mutual information.

• If mutual information is large, there is likely some connection between the two things being looked at.
  • Since temperature and month are connected, their mutual information would be a lot larger than zero.
  • It is not easy to know if mutual information is significant or large.

• If mutual information is one, then knowing one value will exactly tell you the other.
  • For example, if a random desk is selected in a classroom, a teacher can know exactly which student will sit there. Knowing which desk was selected tells us exactly which student is connected to it.

Mutual information does not change based on which of the two random values is revealed. This means we know just as much about temperature when told the month as we know about the month when told the temperature.

Mutual information is difficult to compare. If we calculate mutual information for weather and another value for a card game, the two values cannot easily be compared.

• Information theory