Data scientists and machine learning engineers often encounter imbalanced datasets in real-world applications, where one class has a significantly larger number of instances than others. This can lead to poor model performance and biased predictions. In the context of F1 racing, imbalanced datasets can hinder the development of predictive models for critical aspects of the sport, such as crashes or failures. To address this challenge, I'll demonstrate how to apply advanced techniques like SMOTE, Borderline SMOTE, and ADASYN to handle imbalanced datasets, using the F1 racing dataset from Kaggle as a case study. By the end of this post, you'll be equipped to tackle similar problems in your own projects.
Key Takeaways
- SMOTE, Borderline SMOTE, and ADASYN can effectively handle imbalanced datasets by generating synthetic samples of the minority class.
- Ensemble methods can be used to combine the predictions of multiple models trained on different subsets of the data, further improving model performance.
- Real-world datasets, such as the F1 racing dataset, can be used to demonstrate the effectiveness of these techniques in improving model performance.
The Problem
Imbalanced datasets are a common problem in machine learning, where one class has a significantly larger number of instances than others. This can lead to poor model performance and biased predictions, as the model becomes biased towards the majority class. In the context of F1 racing, imbalanced datasets can hinder the development of predictive models for critical aspects of the sport, such as crashes or failures.
Data and Sources
The F1 racing dataset from Kaggle (https://www.kaggle.com/rohanrao/formula-1-world-championship-1950-2022) contains data on F1 racing teams, drivers, and races from 1950 to 2022. Data accessed on 2024-09-16. This dataset provides a rich source of information for developing predictive models, but it also presents the challenge of imbalanced classes.
Loading the Data
To load the data, we can use the pandas library to read the CSV file.
import pandas as pd
df = pd.read_csv("f1_racing_data.csv")
Step 1 — Data Preparation
In this step, we'll prepare the data for analysis by handling missing values and encoding categorical variables.
import numpy as np
from sklearn.preprocessing import LabelEncoder
# Handle missing values
df.fillna(df.mean(), inplace=True)
# Encode categorical variables
le = LabelEncoder()
df["driver"] = le.fit_transform(df["driver"])
df["team"] = le.fit_transform(df["team"])
Step 2 — SMOTE and Borderline SMOTE
In this step, we'll apply SMOTE and Borderline SMOTE to generate synthetic samples of the minority class.
from imblearn.over_sampling import SMOTE, BorderlineSMOTE
# Apply SMOTE
smote = SMOTE()
X_smote, y_smote = smote.fit_resample(df.drop("target", axis=1), df["target"])
# Apply Borderline SMOTE
borderline_smote = BorderlineSMOTE()
X_borderline_smote, y_borderline_smote = borderline_smote.fit_resample(df.drop("target", axis=1), df["target"])
Step 3 — ADASYN and Ensemble Methods
In this step, we'll apply ADASYN to generate synthetic samples of the minority class and use ensemble methods to combine the predictions of multiple models.
from imblearn.over_sampling import ADASYN
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
# Apply ADASYN
adasyn = ADASYN()
X_adasyn, y_adasyn = adasyn.fit_resample(df.drop("target", axis=1), df["target"])
# Train a random forest classifier on the original data
X_train, X_test, y_train, y_test = train_test_split(df.drop("target", axis=1), df["target"], test_size=0.2, random_state=42)
rf = RandomForestClassifier()
rf.fit(X_train, y_train)
# Train a random forest classifier on the data with synthetic samples
rf_smote = RandomForestClassifier()
rf_smote.fit(X_smote, y_smote)
rf_borderline_smote = RandomForestClassifier()
rf_borderline_smote.fit(X_borderline_smote, y_borderline_smote)
rf_adasyn = RandomForestClassifier()
rf_adasyn.fit(X_adasyn, y_adasyn)
Step 4 — Model Evaluation
In this step, we'll evaluate the performance of the models using metrics such as precision, recall, F1-score, and AUC-ROC.
from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score
# Evaluate the performance of the models
y_pred_rf = rf.predict(X_test)
y_pred_rf_smote = rf_smote.predict(X_test)
y_pred_rf_borderline_smote = rf_borderline_smote.predict(X_test)
y_pred_rf_adasyn = rf_adasyn.predict(X_test)
print("Precision:", precision_score(y_test, y_pred_rf))
print("Recall:", recall_score(y_test, y_pred_rf))
print("F1-score:", f1_score(y_test, y_pred_rf))
print("AUC-ROC:", roc_auc_score(y_test, y_pred_rf))
print("Precision (SMOTE):", precision_score(y_test, y_pred_rf_smote))
print("Recall (SMOTE):", recall_score(y_test, y_pred_rf_smote))
print("F1-score (SMOTE):", f1_score(y_test, y_pred_rf_smote))
print("AUC-ROC (SMOTE):", roc_auc_score(y_test, y_pred_rf_smote))
print("Precision (Borderline SMOTE):", precision_score(y_test, y_pred_rf_borderline_smote))
print("Recall (Borderline SMOTE):", recall_score(y_test, y_pred_rf_borderline_smote))
print("F1-score (Borderline SMOTE):", f1_score(y_test, y_pred_rf_borderline_smote))
print("AUC-ROC (Borderline SMOTE):", roc_auc_score(y_test, y_pred_rf_borderline_smote))
print("Precision (ADASYN):", precision_score(y_test, y_pred_rf_adasyn))
print("Recall (ADASYN):", recall_score(y_test, y_pred_rf_adasyn))
print("F1-score (ADASYN):", f1_score(y_test, y_pred_rf_adasyn))
print("AUC-ROC (ADASYN):", roc_auc_score(y_test, y_pred_rf_adasyn))
Complete Script
The full runnable script combining all steps:
#!/usr/bin/env python3
import pandas as pd
import numpy as np
from sklearn.preprocessing import LabelEncoder
from imblearn.over_sampling import SMOTE, BorderlineSMOTE, ADASYN
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score
# Load the data
df = pd.read_csv("f1_racing_data.csv")
# Handle missing values
df.fillna(df.mean(), inplace=True)
# Encode categorical variables
le = LabelEncoder()
df["driver"] = le.fit_transform(df["driver"])
df["team"] = le.fit_transform(df["team"])
# Apply SMOTE
smote = SMOTE()
X_smote, y_smote = smote.fit_resample(df.drop("target", axis=1), df["target"])
# Apply Borderline SMOTE
borderline_smote = BorderlineSMOTE()
X_borderline_smote, y_borderline_smote = borderline_smote.fit_resample(df.drop("target", axis=1), df["target"])
# Apply ADASYN
adasyn = ADASYN()
X_adasyn, y_adasyn = adasyn.fit_resample(df.drop("target", axis=1), df["target"])
# Train a random forest classifier on the original data
X_train, X_test, y_train, y_test = train_test_split(df.drop("target", axis=1), df["target"], test_size=0.2, random_state=42)
rf = RandomForestClassifier()
rf.fit(X_train, y_train)
# Train a random forest classifier on the data with synthetic samples
rf_smote = RandomForestClassifier()
rf_smote.fit(X_smote, y_smote)
rf_borderline_smote = RandomForestClassifier()
rf_borderline_smote.fit(X_borderline_smote, y_borderline_smote)
rf_adasyn = RandomForestClassifier()
rf_adasyn.fit(X_adasyn, y_adasyn)
# Evaluate the performance of the models
y_pred_rf = rf.predict(X_test)
y_pred_rf_smote = rf_smote.predict(X_test)
y_pred_rf_borderline_smote = rf_borderline_smote.predict(X_test)
y_pred_rf_adasyn = rf_adasyn.predict(X_test)
print("Precision:", precision_score(y_test, y_pred_rf))
print("Recall:", recall_score(y_test, y_pred_rf))
print("F1-score:", f1_score(y_test, y_pred_rf))
print("AUC-ROC:", roc_auc_score(y_test, y_pred_rf))
print("Precision (SMOTE):", precision_score(y_test, y_pred_rf_smote))
print("Recall (SMOTE):", recall_score(y_test, y_pred_rf_smote))
print("F1-score (SMOTE):", f1_score(y_test, y_pred_rf_smote))
print("AUC-ROC (SMOTE):", roc_auc_score(y_test, y_pred_rf_smote))
print("Precision (Borderline SMOTE):", precision_score(y_test, y_pred_rf_borderline_smote))
print("Recall (Borderline SMOTE):", recall_score(y_test, y_pred_rf_borderline_smote))
print("F1-score (Borderline SMOTE):", f1_score(y_test, y_pred_rf_borderline_smote))
print("AUC-ROC (Borderline SMOTE):", roc_auc_score(y_test, y_pred_rf_borderline_smote))
print("Precision (ADASYN):", precision_score(y_test, y_pred_rf_adasyn))
print("Recall (ADASYN):", recall_score(y_test, y_pred_rf_adasyn))
print("F1-score (ADASYN):", f1_score(y_test, y_pred_rf_adasyn))
print("AUC-ROC (ADASYN):", roc_auc_score(y_test, y_pred_rf_adasyn))
Expected Output
The output will show the performance metrics of the models trained on the original data and the data with synthetic samples, including precision, recall, F1-score, and AUC-ROC.
Limitations and Tradeoffs
The techniques presented in this post have their limitations and tradeoffs. For example, SMOTE and Borderline SMOTE can introduce noise in the data, while ADASYN can be computationally expensive. Additionally, the choice of technique depends on the specific problem and dataset. In production, it's essential to carefully evaluate the performance of each technique and select the one that best suits the problem at hand.
Frequently Asked Questions
What is the difference between SMOTE and Borderline SMOTE?
SMOTE generates synthetic samples of the minority class by interpolating between existing samples, while Borderline SMOTE generates synthetic samples near the border of the minority class.
How does ADASYN work?
ADASYN generates synthetic samples of the minority class by adaptively changing the weights of the minority class samples based on their density and proximity to the decision boundary.
Can these techniques be used with other machine learning algorithms?
Yes, these techniques can be used with other machine learning algorithms, such as support vector machines, gradient boosting, and neural networks.
What I'd Change
In conclusion, handling imbalanced datasets is a critical challenge in machine learning, and advanced techniques like SMOTE, Borderline SMOTE, and ADASYN can significantly improve model performance. However, these techniques have their limitations and tradeoffs, and the choice of technique depends on the specific problem and dataset. If I were to redo this project, I would explore other techniques, such as ensemble methods and transfer learning, and evaluate their performance on a larger and more diverse set of datasets. Additionally, I would consider using more advanced metrics, such as the F1-score and AUC-ROC, to evaluate the performance of the models. By doing so, I believe that I can develop even more effective solutions for handling imbalanced datasets in real-world applications.
