SMOTE and Edited Nearest Neighbors Undersampling for Imbalanced Classification

Imbalanced datasets are a special case for classification problem where the class distribution is not uniform among the classes. One of the techniques to handle imbalance datasets is data sampling.

Synthetic Minority Oversampling Technique (SMOTE) is an oversampling technique that generates synthetic samples from the minority class to match the majority class. It is used to obtain a synthetically class-balanced or nearly class-balanced training set. SMOTE works by selecting examples that are close in the feature space, drawing a line between the examples in the feature space and drawing a new sample at a point along that line.

Edited Nearest Neighbor (ENN) is an undersampling method technique that remove the majority class to match the minority class. ENN works by removing samples whose class label differs from the class of the majority of their k nearest neighbors.

Both types of sampling can be used together to handle imbalanced datasets. We can demonstrate this on a simple binary classification problem with a 1:100 imbalance class.

Python
import numpy as np
import matplotlib.pyplot as plt
from collections import Counter
from sklearn.datasets import make_classification

# define dataset
X, y = make_classification(n_samples=10000, n_features=2, n_redundant=0, n_clusters_per_class=1, weights=[0.99], flip_y=0, random_state=3)

# count the class distribution
counter = Counter(y)
print(counter)

# plot the dataset
colors = ["tomato", "mediumspringgreen"]
for label, i in counter.items():
  row_x = np.where(y == label)[0]
  plt.scatter(X[row_x, 0], X[row_x, 1], color=colors[label], label=str(label))
plt.legend()
plt.show()

Scatter plot with 9,900 orange majority samples and 100 green minority samples scattered along the lower edge — The plot without the sampling method

We can fit a DecisionTreeClassifier model on this datasets as the benchmark model.

Python
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold

# define model
model = DecisionTreeClassifier()

# evaluate model
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3)
scores = cross_val_score(model, X, y, scoring='roc_auc', cv=cv)
print('ROC AUC: %.3f' % np.mean(scores))

ROC AUC: 0.737

The model achieved a ROC AUC of about 0.737 without the sampling method.

Now lets try SMOTE and ENN sampling techniques to our example datasets.

Python
from collections import Counter
from imblearn.combine import SMOTEENN

# sampling the data
resample = SMOTEENN()
X, y = oversample.fit_resample(X, y)

# count the class distribution
counter = Counter(y)
print(counter)

# plot the dataset
colors = ["tomato", "mediumspringgreen"]
for label, i in counter.items():
  row_x = np.where(y == label)[0]
  plt.scatter(X[row_x, 0], X[row_x, 1], color=colors[label], label=str(label))
plt.legend()
plt.show()

The same scatter plot after resampling: 9,900 majority samples and 990 minority samples forming a dense green band — The plot after we use the sampling method

Now lets fit our previous DecisionTreeClassifier model on this datasets after we used the sampling method.

Python
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold

# define model
model = DecisionTreeClassifier()

# evaluate model
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
scores = cross_val_score(model, X, y, scoring='roc_auc', cv=cv, n_jobs=-1)
print('Mean ROC AUC: %.3f' % np.mean(scores))

ROC AUC: 0.921

The model achieved a ROC AUC of about 0.921 with the sampling method much better than without sampling. Our results show that the oversampling and undersampling can provided a good results for imbalanced datasets.

#machine-learning#imbalanced-data#data-sampling