Information Gain

An Attribute Selection Method which selects the attribute with the highest information gain.

$\mathrm{Gain}(A)=\mathrm{Info}(D)-{\mathrm{Info}}_A(D)$ where $\text{Info}$ is the Expected Information also called Entropy.

So for a Binary Classification we have $\mathrm{Gain}(A)=I\left(⟨\frac{p}{p+n},\frac{n}{p+n}⟩\right)-{\sum }_a\frac{p_a+n_a}{p+n}I\left(⟨\frac{p_a}{p_a+n_a},\frac{n_a}{p_a+n_a}⟩\right)$

Calculate Expected Information for every attribute and data partition.

def information_partitioned(dataset: pd.DataFrame, target_attribute: str, partition_attribute: str) -> float:
 
    weights = dataset.value_counts(partition_attribute) / dataset.shape[0]
    return sum(
        [
	        weight * information(
	            dataset[dataset[partition_attribute] == index],
	            target_attribute
            )
	        for index, weight in weights.items()
        ]
    )

→ See Expected Information for information()

def information_gain(dataset: pd.DataFrame, target_attribute: str, partition_attribute: str) -> float:
 
    return information(dataset, target_attribute) - information_partitioned(dataset, target_attribute, partition_attribute)

Select attribute with the highest gain first.

Disadvantages

• Biased towards multi-valued attributes
• favors attributes with large numbers