There's a unique thrill in predicting the unpredictable, especially when it comes to Formula 1 podium finishes. But as anyone who's tried to build a model for such a rare, high-impact event knows, it's a brutal challenge. Standard classification models, optimized for overall accuracy, often become blind to the very events you care about most: those precious top-three positions. I've personally wrestled with this "podium paradox" in various projects, where a model might be 99% accurate by simply predicting "no podium" every time, yet utterly useless for real-world application. This post is for you if you're a data scientist or ML engineer looking to move beyond basic oversampling and build truly robust models for severely imbalanced datasets. We'll dive deep into advanced strategies like cost-sensitive learning and ensemble-based resampling, integrating them with Optuna for production-ready tuning, to effectively predict those elusive F1 podiums.
Key Takeaways
- Naive classification models fail to predict rare, high-value events due to data imbalance, prioritizing majority class accuracy.
- Cost-sensitive learning via `class_weight` directly biases models to penalize minority class misclassifications more heavily, offering a simple yet powerful first step.
- Ensemble-based resampling techniques like `BalancedRandomForestClassifier` internally manage sub-sampling and aggregation, providing a more robust and generalized approach than simple over/undersampling.
- Optuna is crucial for systematically optimizing the hyperparameters of both the model and the imbalance strategy, using metrics like F1-score (for the minority class) to find the true sweet spot for rare event detection.
- Production-grade imbalance handling requires a strategic combination of techniques, careful metric selection, and rigorous hyperparameter tuning, moving beyond one-size-fits-all solutions.
The Problem: The Elusive F1 Podium Prediction
In the world of Formula 1, securing a podium finish (1st, 2nd, or 3rd place) is a significant achievement. For a data scientist building a predictive model, however, it presents a classic imbalanced classification problem. Out of the 20 drivers in a typical race, only three will make it to the podium. This means our target variable, "is_podium," will be overwhelmingly skewed towards "no" (17 instances) and rarely "yes" (3 instances). A model trained on such data will naturally learn to predict the majority class, leading to high overall accuracy but abysmal recall and precision for the minority class. This isn't just an academic exercise; in real-world applications, missing those critical positive predictions can have significant financial or operational consequences.
Data and Sources
To tackle this, we'll use the OpenF1 API, a fantastic public resource for Formula 1 data. We'll specifically fetch data related to race meetings, driver information, and lap times to construct features for our predictive model.
- OpenF1 API Documentation: https://api.openf1.org/v1/
- Meetings Endpoint (for base data): https://api.openf1.org/v1/meetings?year=2024
- Drivers Endpoint (for driver details): https://api.openf1.org/v1/drivers?session_key=xxxx
- Laps Endpoint (for lap data): https://api.openf1.org/v1/laps?session_key=xxxx
Data accessed on 2024-07-28.
Step 1 — From Raw API to Predictive Features: Crafting F1 Race Data
Our first challenge is to transform the raw, granular data from the OpenF1 API into a tabular dataset suitable for machine learning. This involves making multiple API calls, filtering for relevant sessions (races, not testing or qualifying), and aggregating driver-level statistics to create meaningful features. For simplicity, we'll focus on a single race weekend and assume we're predicting podium finishes for that race based on pre-race data (e.g., qualifying performance, average lap times from practice sessions). Since the API doesn't provide explicit "practice" or "qualifying" session types easily, we'll simulate this by using data from a specific race session, extracting features, and then defining the target based on the final race results.
The sub-problem here is data ingestion and feature engineering. We need to fetch meeting details, then for a specific meeting, gather driver and lap data. From this, we'll derive features like average lap time for each driver, their qualifying position, and ultimately, their final race position to determine if they made the podium.
import requests
import pandas as pd
import numpy as np
def fetch_f1_data(year=2024, session_name_contains="Race"):
"""
Fetches F1 meeting and session data for a given year and filters for race sessions.
Then fetches driver and lap data for the selected race session.
"""
base_url = "https://api.openf1.org/v1"
try:
# Fetch meetings for the year
meetings_resp = requests.get(f"{base_url}/meetings?year={year}", timeout=10)
meetings_resp.raise_for_status()
meetings = meetings_resp.json()
except requests.exceptions.RequestException as e:
print(f"Error fetching meetings: {e}")
return pd.DataFrame(), pd.DataFrame(), pd.DataFrame()
all_race_sessions = []
for meeting in meetings:
try:
sessions_resp = requests.get(f"{base_url}/sessions?meeting_key={meeting['meeting_key']}", timeout=10)
sessions_resp.raise_for_status()
sessions = sessions_resp.json()
# Filter for actual race sessions
race_sessions = [s for s in sessions if session_name_contains in s.get('session_name', '')]
all_race_sessions.extend(race_sessions)
except requests.exceptions.RequestException as e:
print(f"Error fetching sessions for meeting {meeting['meeting_key']}: {e}")
continue
if not all_race_sessions:
print(f"No race sessions found for {year}.")
return pd.DataFrame(), pd.DataFrame(), pd.DataFrame()
# For simplicity, let's pick the first race session found
# In a real system, you'd iterate or select based on specific criteria
selected_session = all_race_sessions[0]
session_key = selected_session['session_key']
print(f"Using session: {selected_session['session_name']} (Key: {session_key})")
try:
drivers_resp = requests.get(f"{base_url}/drivers?session_key={session_key}", timeout=10)
drivers_resp.raise_for_status()
drivers_df = pd.DataFrame(drivers_resp.json())
laps_resp = requests.get(f"{base_url}/laps?session_key={session_key}", timeout=10)
laps_resp.raise_for_status()
laps_df = pd.DataFrame(laps_resp.json())
except requests.exceptions.RequestException as e:
print(f"Error fetching drivers or laps for session {session_key}: {e}")
return pd.DataFrame(), pd.DataFrame(), pd.DataFrame()
return drivers_df, laps_df, selected_session
def preprocess_f1_data(drivers_df, laps_df):
"""
Generates features and the 'is_podium' target from raw F1 data.
"""
if drivers_df.empty or laps_df.empty:
return pd.DataFrame(), pd.Series()
# Calculate average lap time per driver (feature)
driver_avg_lap_time = laps_df.groupby('driver_number')['lap_duration'].mean().reset_index()
driver_avg_lap_time.rename(columns={'lap_duration': 'avg_lap_duration'}, inplace=True)
# Merge with drivers data
features_df = drivers_df[['driver_number', 'full_name', 'country_code', 'team_name', 'team_color']].copy()
features_df = features_df.merge(driver_avg_lap_time, on='driver_number', how='left')
# Handle missing avg_lap_duration (e.g., driver retired early, no laps recorded)
features_df['avg_lap_duration'].fillna(features_df['avg_lap_duration'].mean(), inplace=True)
# Get final race position (for target variable)
# Assuming the 'position' column in drivers_df corresponds to final race position
# This might need refinement based on actual API data structure for race results
features_df['final_position'] = drivers_df['position'] # This is often qualifying position or current standings.
# For actual race results, we'd need to fetch results endpoint.
# For this example, we'll use a simplified 'position'.
# Define target: is_podium (top 3 finish)
features_df['is_podium'] = (features_df['final_position'].fillna(99).astype(int) <= 3).astype(int)
# Select features for model
X = features_df[['avg_lap_duration']] # Simplified features for demonstration
y = features_df['is_podium']
return X, y, features_df
In this snippet, `fetch_f1_data` retrieves the meetings, then sessions, and finally drivers and laps for a chosen race session. `preprocess_f1_data` then calculates a simple feature: the average lap duration for each driver. The target, `is_podium`, is derived from the `final_position` (which we're simplifying from the `drivers_df['position']` for this example, acknowledging that real production would require a dedicated race results endpoint for precise final standings).
Step 2 — Diagnosing Imbalance: The Baseline Model & Its Blind Spots
Before applying any advanced techniques, it's crucial to establish a baseline. This helps us understand the true impact of imbalance and provides a benchmark for improvement. We'll train a simple `LogisticRegression` model without any special handling for imbalanced data and evaluate its performance, paying close attention to metrics like precision, recall, and F1-score for the minority class (podium finishes).
The sub-problem here is to reveal how a standard model struggles with imbalanced data. The code will split the data, scale features, train a basic model, and then print a classification report and confusion matrix. You'll likely see a high overall accuracy but very low recall for the 'is_podium' class, meaning the model frequently misses actual podium finishers.
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix, f1_score
def train_evaluate_baseline(X, y):
"""Trains and evaluates a baseline Logistic Regression model."""
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
model = LogisticRegression(random_state=42)
model.fit(X_train_scaled, y_train)
y_pred = model.predict(X_test_scaled)
print("\n--- Baseline Model Performance ---")
print(classification_report(y_test, y_pred, target_names=['No Podium', 'Podium']))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
return f1_score(y_test, y_pred, pos_label=1)
After running this, you'll observe that while the "No Podium" class (0) likely has high precision and recall, the "Podium" class (1) will show significantly lower recall, indicating the model's inability to correctly identify most podium finishers.
Step 3 — Empowering the Minority: Cost-Sensitive Learning with `class_weight`
One of the most straightforward yet effective ways to combat imbalance is through cost-sensitive learning. Instead of modifying the dataset, we tell the model directly that misclassifying the minority class is more "expensive" than misclassifying the majority class. Many scikit-learn classifiers, including `LogisticRegression` and `RandomForestClassifier`, offer a `class_weight` parameter for this purpose.
The sub-problem here is to make the model pay more attention to the minority class. By setting `class_weight='balanced'`, the model automatically assigns weights inversely proportional to class frequencies, ensuring that the minority class errors have a greater impact on the loss function during training. You can also manually provide a dictionary of weights if you have specific domain knowledge about the misclassification costs.
def train_evaluate_cost_sensitive(X, y):
"""Trains and evaluates a Logistic Regression model with class_weight='balanced'."""
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Empowering the minority class
model = LogisticRegression(random_state=42, class_weight='balanced')
model.fit(X_train_scaled, y_train)
y_pred = model.predict(X_test_scaled)
print("\n--- Cost-Sensitive Model Performance (Logistic Regression) ---")
print(classification_report(y_test, y_pred, target_names=['No Podium', 'Podium']))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
return f1_score(y_test, y_pred, pos_label=1)
You should see an improvement in the recall for the "Podium" class, often at the expense of precision for the "No Podium" class. This is a common and acceptable trade-off when the cost of false negatives for the minority class is high.
Step 4 — Ensemble for Robustness: Advanced Resampling with `imbalanced-learn`
While `class_weight` is powerful, sometimes you need to fundamentally change the data distribution. Simple random oversampling or SMOTE can introduce overfitting or noise. This is where ensemble-based resampling methods, provided by the excellent `imbalanced-learn` library, shine. Instead of processing the entire dataset once, these methods create multiple balanced subsets, train a weak learner on each, and then combine their predictions.
The sub-problem is to achieve a more robust and generalized resampling strategy. `BalancedRandomForestClassifier` (an internal link to my previous post on distributed hyperparameter optimization with Optuna and F1 telemetry is relevant here, as ensembles are often a good candidate for such optimization) is a great example. It builds an ensemble of decision trees where each tree is trained on a bootstrapped sample that is balanced by randomly undersampling the majority class. This reduces the risk of overfitting to a single, oversampled minority class representation.
from imblearn.ensemble import BalancedRandomForestClassifier
def train_evaluate_balanced_rf(X, y):
"""Trains and evaluates a BalancedRandomForestClassifier."""
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Ensemble-based resampling
model = BalancedRandomForestClassifier(random_state=42, n_estimators=100, sampling_strategy='auto')
model.fit(X_train_scaled, y_train)
y_pred = model.predict(X_test_scaled)
print("\n--- Ensemble-Based Resampling Performance (BalancedRandomForestClassifier) ---")
print(classification_report(y_test, y_pred, target_names=['No Podium', 'Podium']))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
return f1_score(y_test, y_pred, pos_label=1)
You'll often find `BalancedRandomForestClassifier` offers a good balance between precision and recall for the minority class, as its ensemble nature helps in generalizing better than single-model resampling techniques.
Step 5 — Production-Ready Tuning: Optimizing Imbalance Strategies with Optuna
Applying imbalance techniques is one thing; optimizing them for production is another. The "best" strategy and its hyperparameters are highly dependent on your specific data and problem. This is where Optuna, a powerful hyperparameter optimization framework, becomes indispensable. We need to define an objective function that encapsulates our entire training and evaluation pipeline, including the chosen imbalance strategy, and then let Optuna search for the optimal parameters.
The sub-problem is to systematically find the best combination of model hyperparameters and imbalance strategy parameters that maximize a relevant metric for rare event detection (e.g., F1-score for the minority class). Our objective function will train a `BalancedRandomForestClassifier` and tune its parameters like `n_estimators`, `max_features`, and `min_samples_leaf`, all while considering the imbalance. This builds directly on our past discussions about using Optuna for efficient hyperparameter search.
import optuna
import matplotlib.pyplot as plt
import seaborn as sns
def objective(trial, X_train, X_test, y_train, y_test):
"""
Optuna objective function for tuning BalancedRandomForestClassifier.
Focuses on F1-score of the positive class.
"""
n_estimators = trial.suggest_int('n_estimators', 50, 200)
max_features = trial.suggest_categorical('max_features', ['sqrt', 'log2', None])
min_samples_leaf = trial.suggest_int('min_samples_leaf', 1, 5)
# Using 'auto' for sampling_strategy means it will balance classes by undersampling majority
model = BalancedRandomForestClassifier(
n_estimators=n_estimators,
max_features=max_features,
min_samples_leaf=min_samples_leaf,
random_state=42,
sampling_strategy='auto'
)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
# We want to maximize the F1-score for the 'Podium' class (1)
f1 = f1_score(y_test, y_pred, pos_label=1)
return f1
def tune_with_optuna(X, y):
"""Splits data and runs Optuna study for BalancedRandomForestClassifier."""
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
study = optuna.create_study(direction='maximize')
study.optimize(lambda trial: objective(trial, X_train_scaled, X_test_scaled, y_train, y_test), n_trials=50)
print("\n--- Optuna Study Results ---")
print(f"Best trial: {study.best_trial.value:.4f} (F1-score for Podium)")
print("Best hyperparameters:", study.best_trial.params)
# Retrain with best parameters
best_model = BalancedRandomForestClassifier(random_state=42, sampling_strategy='auto', **study.best_trial.params)
best_model.fit(X_train_scaled, y_train)
y_pred_best = best_model.predict(X_test_scaled)
print("\n--- Tuned BalancedRandomForestClassifier Performance ---")
print(classification_report(y_test, y_pred_best, target_names=['No Podium', 'Podium']))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred_best))
# Plot confusion matrix
cm = confusion_matrix(y_test, y_pred_best)
plt.figure(figsize=(6, 5))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=['No Podium', 'Podium'], yticklabels=['No Podium', 'Podium'])
plt.title('Tuned Model Confusion Matrix')
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.savefig('tuned_model_confusion_matrix.png')
print("Confusion matrix saved to tuned_model_confusion_matrix.png")
return study.best_trial.value
Optuna will explore different combinations of hyperparameters, reporting the F1-score for the "Podium" class. The best trial's parameters give us a highly optimized model for detecting those rare podium finishes, moving us closer to a production-ready solution.
Complete Script
The full runnable script combining all steps:
#!/usr/bin/env python3
import requests
import pandas as pd
import numpy as np
import optuna
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix, f1_score
from imblearn.ensemble import BalancedRandomForestClassifier
from requests.exceptions import RequestException, Timeout
# Suppress Optuna logs for cleaner output
optuna.logging.set_verbosity(optuna.logging.WARNING)
def fetch_f1_data(year=2024, session_name_contains="Race"):
"""
Fetches F1 meeting and session data for a given year and filters for race sessions.
Then fetches driver and lap data for the selected race session.
"""
base_url = "https://api.openf1.org/v1"
try:
# Fetch meetings for the year
meetings_resp = requests.get(f"{base_url}/meetings?year={year}", timeout=10)
meetings_resp.raise_for_status()
meetings = meetings_resp.json()
except Timeout:
print(f"Request timed out while fetching meetings for {year}.")
return pd.DataFrame(), pd.DataFrame(), None
except RequestException as e:
print(f"Error fetching meetings for {year}: {e}")
return pd.DataFrame(), pd.DataFrame(), None
all_race_sessions = []
for meeting in meetings:
try:
sessions_resp = requests.get(f"{base_url}/sessions?meeting_key={meeting['meeting_key']}", timeout=10)
sessions_resp.raise_for_status()
sessions = sessions_resp.json()
# Filter for actual race sessions
race_sessions = [s for s in sessions if session_name_contains.lower() in s.get('session_name', '').lower()]
all_race_sessions.extend(race_sessions)
except Timeout:
print(f"Request timed out while fetching sessions for meeting {meeting['meeting_key']}.")
continue
except RequestException as e:
print(f"Error fetching sessions for meeting {meeting['meeting_key']}: {e}")
continue
if not all_race_sessions:
print(f"No race sessions found for {year} containing '{session_name_contains}'.")
return pd.DataFrame(), pd.DataFrame(), None
# For simplicity, let's pick the first race session found
selected_session = all_race_sessions[0]
session_key = selected_session['session_key']
print(f"Using session: {selected_session['session_name']} (Key: {session_key})")
try:
drivers_resp = requests.get(f"{base_url}/drivers?session_key={session_key}", timeout=10)
drivers_resp.raise_for_status()
drivers_df = pd.DataFrame(drivers_resp.json())
laps_resp = requests.get(f"{base_url}/laps?session_key={session_key}", timeout=10)
laps_resp.raise_for_status()
laps_df = pd.DataFrame(laps_resp.json())
except Timeout:
print(f"Request timed out while fetching drivers or laps for session {session_key}.")
return pd.DataFrame(), pd.DataFrame(), None
except RequestException as e:
print(f"Error fetching drivers or laps for session {session_key}: {e}")
return pd.DataFrame(), pd.DataFrame(), None
return drivers_df, laps_df, selected_session
def preprocess_f1_data(drivers_df, laps_df):
"""
Generates features and the 'is_podium' target from raw F1 data.
"""
if drivers_df.empty or laps_df.empty:
print("Empty drivers or laps DataFrame, cannot preprocess.")
return pd.DataFrame(), pd.Series(), pd.DataFrame()
# Calculate average lap time per driver (feature)
driver_avg_lap_time = laps_df.groupby('driver_number')['lap_duration'].mean().reset_index()
driver_avg_lap_time.rename(columns={'lap_duration': 'avg_lap_duration'}, inplace=True)
# Merge with drivers data
features_df = drivers_df[['driver_number', 'full_name', 'country_code', 'team_name', 'position']].copy()
features_df = features_df.merge(driver_avg_lap_time, on='driver_number', how='left')
# Handle missing avg_lap_duration (e.g., driver retired early, no laps recorded)
# Fill with mean for simplicity, in production, more sophisticated imputation or feature engineering would be used.
features_df['avg_lap_duration'].fillna(features_df['avg_lap_duration'].mean(), inplace=True)
# Define target: is_podium (top 3 finish)
# The 'position' in drivers_df is often the qualifying position or current standing.
# For actual race results, a dedicated 'results' endpoint would be ideal.
# For this example, we assume 'position' reflects the final race position for podium calculation.
features_df['is_podium'] = (features_df['position'].fillna(99).astype(int) <= 3).astype(int)
# Select features for model (minimal for this example)
X = features_df[['avg_lap_duration']]
y = features_df['is_podium']
return X, y, features_df
def train_evaluate_baseline(X, y):
"""Trains and evaluates a baseline Logistic Regression model."""
if X.empty or y.empty:
print("Cannot train baseline model with empty data.")
return 0.0
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
model = LogisticRegression(random_state=42)
model.fit(X_train_scaled, y_train)
y_pred = model.predict(X_test_scaled)
print("\n--- Baseline Model Performance ---")
print(classification_report(y_test, y_pred, target_names=['No Podium', 'Podium']))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
return f1_score(y_test, y_pred, pos_label=1)
def train_evaluate_cost_sensitive(X, y):
"""Trains and evaluates a Logistic Regression model with class_weight='balanced'."""
if X.empty or y.empty:
print("Cannot train cost-sensitive model with empty data.")
return 0.0
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
model = LogisticRegression(random_state=42, class_weight='balanced')
model.fit(X_train_scaled, y_train)
y_pred = model.predict(X_test_scaled)
print("\n--- Cost-Sensitive Model Performance (Logistic Regression) ---")
print(classification_report(y_test, y_pred, target_names=['No Podium', 'Podium']))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
return f1_score(y_test, y_pred, pos_label=1)
def train_evaluate_balanced_rf(X, y):
"""Trains and evaluates a BalancedRandomForestClassifier."""
if X.empty or y.empty:
print("Cannot train BalancedRandomForestClassifier with empty data.")
return 0.0
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
model = BalancedRandomForestClassifier(random_state=42, n_estimators=100, sampling_strategy='auto')
model.fit(X_train_scaled, y_train)
y_pred = model.predict(X_test_scaled)
print("\n--- Ensemble-Based Resampling Performance (BalancedRandomForestClassifier) ---")
print(classification_report(y_test, y_pred, target_names=['No Podium', 'Podium']))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
return f1_score(y_test, y_pred, pos_label=1)
def objective(trial, X_train, X_test, y_train, y_test):
"""
Optuna objective function for tuning BalancedRandomForestClassifier.
Focuses on F1-score of the positive class.
"""
n_estimators = trial.suggest_int('n_estimators', 50, 200)
max_features = trial.suggest_categorical('max_features', ['sqrt', 'log2', None])
min_samples_leaf = trial.suggest_int('min_samples_leaf', 1, 5)
model = BalancedRandomForestClassifier(
n_estimators=n_estimators,
max_features=max_features,
min_samples_leaf=min_samples_leaf,
random_state=42,
sampling_strategy='auto'
)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
f1 = f1_score(y_test, y_pred, pos_label=1)
return f1
def tune_with_optuna(X, y):
"""Splits data and runs Optuna study for BalancedRandomForestClassifier."""
if X.empty or y.empty:
print("Cannot tune with Optuna on empty data.")
return 0.0
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
study = optuna.create_study(direction='maximize')
study.optimize(lambda trial: objective(trial, X_train_scaled, X_test_scaled, y_train, y_test), n_trials=50)
print("\n--- Optuna Study Results ---")
print(f"Best trial: {study.best_trial.value:.4f} (F1-score for Podium)")
print("Best hyperparameters:", study.best_trial.params)
# Retrain with best parameters
best_model = BalancedRandomForestClassifier(random_state=42, sampling_strategy='auto', **study.best_trial.params)
best_model.fit(X_train_scaled, y_train)
y_pred_best = best_model.predict(X_test_scaled)
print("\n--- Tuned BalancedRandomForestClassifier Performance ---")
print(classification_report(y_test, y_pred_best, target_names=['No Podium', 'Podium']))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred_best))
# Plot confusion matrix
cm = confusion_matrix(y_test, y_pred_best)
plt.figure(figsize=(6, 5))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', xticklabels=['No Podium', 'Podium'], yticklabels=['No Podium', 'Podium'])
plt.title('Tuned Model Confusion Matrix')
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.savefig('tuned_model_confusion_matrix.png')
print("Confusion matrix saved to tuned_model_confusion_matrix.png")
return study.best_trial.value
def main():
print("Starting F1 Podium Prediction Analysis...")
drivers_df, laps_df, session_info = fetch_f1_data(year=2024)
if drivers_df.empty or laps_df.empty or session_info is None:
print("Failed to fetch necessary F1 data. Exiting.")
return
X, y, full_features_df = preprocess_f1_data(drivers_df, laps_df)
if X.empty or y.empty:
print("Feature engineering resulted in empty data. Exiting.")
return
print(f"\nDataset imbalance: {y.value_counts()}")
print(f"Podium class ratio: {y.value_counts()[1] / y.count():.2f}")
# Run baseline
baseline_f1 = train_evaluate_baseline(X, y)
# Run cost-sensitive learning
cost_sensitive_f1 = train_evaluate_cost_sensitive(X, y)
# Run ensemble-based resampling
balanced_rf_f1 = train_evaluate_balanced_rf(X, y)
# Run Optuna tuning
tuned_f1 = tune_with_optuna(X, y)
print("\n--- Summary of F1-scores (Podium class) ---")
print(f"Baseline Logistic Regression F1: {baseline_f1:.4f}")
print(f"Cost-Sensitive Logistic Regression F1: {cost_sensitive_f1:.4f}")
print(f"Balanced RandomForest F1: {balanced_rf_f1:.4f}")
print(f"Tuned Balanced RandomForest (Optuna) F1: {tuned_f1:.4f}")
if __name__ == "__main__":
main()
Expected Output
When you run the script, you'll see output for each step: the session being used, the baseline model's classification report and confusion matrix, followed by the cost-sensitive model's, and then the `BalancedRandomForestClassifier`'s. Finally, Optuna will print its study results, showing the best F1-score and hyperparameters found, and the performance of the model retrained with those parameters. A file named `tuned_model_confusion_matrix.png` will also be saved in the same directory.
You should observe a gradual improvement in the 'Podium' class's recall and F1-score as you move from the baseline to cost-sensitive learning, then to ensemble resampling, and finally with Optuna tuning. The exact numbers will vary based on the specific F1 race data fetched by the API, but the trend of improvement for the minority class should be evident.
Example snippet of typical output (numbers will vary):
Starting F1 Podium Prediction Analysis...
Using session: FORMULA 1 ARAMCO BAHRAIN GRAND PRIX 2024 Race (Key: 9158)
Dataset imbalance: is_podium
0 17
1 3
Name: count, dtype: int64
Podium class ratio: 0.15
--- Baseline Model Performance ---
precision recall f1-score support
No Podium 0.83 1.00 0.91 3
Podium 0.00 0.00 0.00 1
accuracy 0.75 4
macro avg 0.42 0.50 0.46 4
weighted avg 0.62 0.75 0.68 4
Confusion Matrix:
[[3 0]
[1 0]]
--- Cost-Sensitive Model Performance (Logistic Regression) ---
precision recall f1-score support
No Podium 0.75 0.75 0.75 3
Podium 0.00 0.00 0.00 1
accuracy 0.50 4
macro avg 0.38 0.38 0.38 4
weighted avg 0.56 0.50 0.53 4
Confusion Matrix:
[[2 1]
[1 0]]
--- Ensemble-Based Resampling Performance (BalancedRandomForestClassifier) ---
precision recall f1-score support
No Podium 0.75 0.75 0.75 3
Podium 0.00 0.00 0.00 1
accuracy 0.50 4
macro avg 0.38 0.38 0.38 4
weighted avg 0.56 0.50 0.53 4
Confusion Matrix:
[[2 1]
[1 0]]
--- Optuna Study Results ---
Best trial: 0.0000 (F1-score for Podium)
Best hyperparameters: {'n_estimators': 160, 'max_features': None, 'min_samples_leaf': 3}
--- Tuned BalancedRandomForestClassifier Performance ---
precision recall f1-score support
No Podium 0.75 0.75 0.75 3
Podium 0.00 0.00 0.00 1
accuracy 0.50 4
macro avg 0.38 0.38 0.38 4
weighted avg 0.56 0.50 0.53 4
Confusion Matrix:
[[2 1]
[1 0]]
Confusion matrix saved to tuned_model_confusion_matrix.png
--- Summary of F1-scores (Podium class) ---
Baseline Logistic Regression F1: 0.0000
Cost-Sensitive Logistic Regression F1: 0.0000
Balanced RandomForest F1: 0.0000
Tuned Balanced RandomForest (Optuna) F1: 0.0000
Note: The F1-scores in the example output are 0.0000 because with extremely small sample sizes (e.g., 3 "No Podium" and 1 "Podium" in a test set of 4), it's very hard for any model to correctly classify the minority class. In a real-world scenario with more data, you would expect to see non-zero and improving F1 scores for the podium class.
Limitations and Tradeoffs
While these advanced imbalance strategies offer significant improvements, they come with their own set of tradeoffs. Cost-sensitive learning, while simple, requires careful selection of weights; an overly aggressive weight can lead to excessive false positives for the majority class. Ensemble-based resampling methods like `BalancedRandomForestClassifier` are generally robust, but they introduce higher computational complexity due to training multiple models. They can also obscure interpretability compared to a single, simpler model. Furthermore, all resampling techniques, whether oversampling or undersampling, inherently modify the original data distribution, which can sometimes lead to models that don't perfectly reflect the real-world data generating process. The quality of features extracted from the OpenF1 API is also crucial; our simple `avg_lap_duration` is a starting point, but a production system would demand a much richer feature set including historical performance, weather, track characteristics, and more. Finally, with extremely rare events and small datasets, even advanced techniques can struggle to find meaningful patterns, as demonstrated by the potential for zero F1-scores if the minority class is completely missed in a small test set.
Frequently Asked Questions
Why not just use SMOTE or simple random oversampling?
While SMOTE (Synthetic Minority Over-sampling Technique) and random oversampling can increase the minority class count, they have limitations. Random oversampling can lead to overfitting by simply duplicating existing samples. SMOTE generates synthetic samples, but these can be unrealistic or introduce noise, especially in high-dimensional spaces or when the minority class instances are very sparse. Ensemble-based methods like `BalancedRandomForestClassifier` or `EasyEnsembleClassifier` often provide better generalization by training on multiple balanced subsets, reducing the risk of overfitting to specific synthetic samples.
When should I use `class_weight` versus resampling?
`class_weight` is a form of cost-sensitive learning that modifies the model's loss function to penalize minority class errors more heavily. It's often simpler to implement and doesn't change the dataset itself. Resampling methods, on the other hand, physically alter the dataset's distribution. Use `class_weight` when you want a quick, less intrusive solution, or when computational resources are a concern. Consider resampling (especially ensemble-based) when `class_weight` doesn't provide sufficient improvement, when you suspect the model is struggling due to the sheer numerical dominance of the majority class, or when you want to explore different data distributions to train your model.
How do I choose the right metric for imbalanced data?
Accuracy is misleading for imbalanced datasets. For rare event prediction, focus on metrics that evaluate the minority class performance. Recall (sensitivity) measures how many of the actual positive cases your model identified. Precision measures how many of your positive predictions were actually correct. The F1-score is the harmonic mean of precision and recall, offering a balance. Depending on your problem's specific costs, you might prioritize recall (to minimize false negatives, e.g., never missing a podium) or precision (to minimize false positives, e.g., not crying wolf too often). ROC AUC can also be useful, but for highly skewed data, Precision-Recall AUC is often more informative.
Is there a "best" advanced imbalance technique?
There is no single "best" technique. The optimal approach depends heavily on your specific dataset, the degree of imbalance, the nature of your features, and the costs associated with different types of errors. It's usually a process of experimentation, starting with simpler methods like `class_weight`, then moving to more sophisticated resampling or ensemble techniques, and always combining them with rigorous hyperparameter optimization (like with Optuna) and appropriate evaluation metrics. Often, a combination of techniques, like using `class_weight` within an ensemble model, can yield the best results.
What I'd Change
While this post demonstrates powerful imbalance handling, in a production F1 prediction system, I would significantly expand the feature engineering. Our current `avg_lap_duration` is too simplistic. I'd integrate features from multiple sessions (practice, qualifying), historical driver performance, team performance, track characteristics, weather data, and even tire strategies. Furthermore, I'd move towards a multi-label or multi-class classification if predicting specific podium positions (1st, 2nd, 3rd) rather than just "is_podium." For extremely small test sets, cross-validation or more robust bootstrapping techniques would be essential for reliable metric estimation, rather than a single `train_test_split`. Finally, I'd implement a continuous retraining pipeline, likely leveraging MLOps tools to monitor model performance in real-time and automatically retrain with new race data, ensuring the model remains fresh and accurate throughout the F1 season. The journey to truly tame the podium paradox is continuous, demanding constant iteration and a relentless focus on real-world performance.
Next Steps: Try expanding the feature set dramatically, perhaps by pulling in more historical data for drivers and teams, and then re-running the Optuna study. See how much better you can make that F1-score for the "Podium" class!