CLASSIFICATION · BEGINNER GUIDE
Class Imbalance: Why Precision and Recall Matter
Understand why accuracy can mislead when positives are rare and how precision, recall and a baseline help.
Quick answer: Class imbalance means one target class is much rarer than another. Accuracy can look high when a model mostly predicts the common class, so inspect class-specific precision, recall and the baseline prevalence.
What this method answers
Precision asks what fraction of predicted positives are truly positive. Recall asks what fraction of actual positives were found. The preferred trade-off depends on whether false alarms or missed cases are more costly.
Use a stratified split when appropriate so rare classes appear in evaluation folds, while avoiding leakage across related records. Report the class counts and assess the intended threshold.
Before running a calculation, write down the population and the exact quantity you want to estimate. A method name alone does not define the question, the comparison or the assumptions.
Steps to follow
- Count positive and negative cases in each split.
- Calculate the majority-class baseline before evaluating a model.
- Report positive-class precision, recall and a threshold-dependent confusion matrix.
- Evaluate final performance on untouched data with realistic class prevalence.
Keep a record of exclusions, recoding decisions and software settings. That makes the analysis easier to check and repeat.
Worked example
A dataset has 1,000 transactions, 20 of which are fraudulent. A model that predicts every transaction as legitimate is 98% accurate.
| Measure for all-negative model | Value |
|---|---|
| Accuracy | 980 / 1,000 = 98% |
| Fraud recall | 0 / 20 = 0% |
| Fraud precision | Undefined (no positive predictions) |
The baseline accuracy is 980/1,000 = 0.98, but the model catches no fraud. The score hides complete failure on the rare class.
Interpret the result carefully
Compare with a simple prevalence baseline and inspect precision-recall behaviour. If false negatives are expensive, recall may matter; if investigations are limited, precision may matter more.
Limit to keep in mind: Resampling or class weights can change probability calibration. Evaluate on a test set with the natural target prevalence and recalibrate if the probabilities guide decisions.
When reporting an analysis, include the sample size, estimate, units and uncertainty. Explain why the method fits the sampling and measurement design. Separate planned tests from exploratory checks, and avoid saying that association proves a cause.
Frequently asked questions
Is 98% accuracy good here?
Not for finding fraud: the all-negative model reaches 98% while missing every positive case.
Should I use oversampling?
It can help training in some settings, but perform it only inside training folds and evaluate on natural-prevalence data.
Which is more important, precision or recall?
It depends on the cost of false positives versus false negatives and available review capacity.
Can class weights change probabilities?
Yes. They can alter the fit and calibration, so check probability reliability if using probabilities.