Naive Bayes Classifier

A simpler Bayesian Classifier which assumes independent attributes / classes which is why we can write $\mathrm{P}\left(\text{ c }\mid {\text{ e }}_1,\dots ,{\text{ e }}_n\right)=\alpha ⟨{\text{ e }}_1,\dots ,{\text{ e }}_n⟩\cdot \mathrm{P}(\text{ c })\cdot {\prod }_i\mathrm{P}\left({\text{ e }}_{\mathbf{i}}\mid \text{ c }\right)$ Assuming boolean variables (which we can always achieve via Binning and One-Hot-Encoding), we have the following parameters in the CPTs $\theta =P(c=\mathrm{T}),{\theta }_{i1}=P\left(X_i=\mathrm{T}\mid C=\mathrm{T}\right),\text{ and }{\theta }_{i2}=P\left(X_i=\mathrm{T}\mid C=\mathrm{F}\right)$

We can then do a prediction with $\mathrm{P}\left(C\mid x_1,\dots ,x_n\right)=\alpha \cdot \mathrm{P}(C)\cdot {\prod }_i\mathrm{P}\left(x_i\mid C\right)$ The cool thing is, that there are only $2n+1$ parameters and there is no search required. Naive Bayes models perform great, even when the data is noisy.

Calculation of Probabilities

If attribute is categorical: $P(x_k|C_i)$ is the number of tuples in class $C_i$ having value $x_k$ divided by the number of tuples of the class $C_i$.

If attribute is continous: Usually computed based on Normal Distribution with $P(x_k|C_i)=G(x_k,{\mu }_{C_i},{\sigma }_{C_i})$

Pros

• easy to implement
• but still good results

Cons

• assumption leads to loss of Accuracy
• as there will always be some kind of dependency amongst variables
  • → Bayesian Belief Network

Implementation

Use this function to test whether datapoint is more likely for class 1 vs. class 0 by estimating loc and scale from the data for the respective classes.

scipy.stats.norm.pdf(value, loc, scale)