Expected Information

$\mathrm{Info}(D)=-{\sum }_{i=1}^m{p}_i{\log }_2\left(p_i\right)$

where $p_i$ is the probability that tuple in $D$ belongs to class $C_i$ which can be estimated by $\frac{|C_i|}{|D|}$. The Expectation of Surprisal over every possible outome.

It is used to calculate Information Gain and Perplexity.

Python Implementation

def information(dataset: pd.DataFrame, target_attribute: str) -> float:
	p = dataset.value_counts(target_attribute) / dataset.shape[0]
	return -sum([pi * log(pi, 2) for pi in p])