الصفحة الرئيسية
data-science
machine-learning

How Bagging, Boosting, and Stacking Differ on Real‑World User Data

Mausam

You have already played with single‑model ensembles like Isolation Forest and wonder whether the extra engineering of a stacking pipeline is justified for a binary classification problem on realistic user records. The answer is yes—if you control overfitting and latency, stacking can squeeze out a measurable accuracy lift over both bagging (RandomForest) and boosting (GradientBoosting) on a modest‑size, publicly available dataset.

Key Takeaways

  • Bagging (RandomForest) gives a strong baseline with low variance but limited ability to capture complex interactions.
  • Boosting (GradientBoosting) reduces bias by sequentially focusing on hard examples, yet can over‑fit small noisy datasets.
  • Stacking combines the complementary strengths of bagging and boosting; when tuned, it often outperforms either alone on modest tabular data.
  • Managing overfitting (e.g., using cross‑validation, simple meta‑learner) and monitoring latency are essential for production viability.
  • For very small or highly imbalanced problems, the extra complexity of stacking may not be worth the marginal gain.

Data and Sources

The experiment uses 500 user records fetched from the public Random User API:

Source reviewed on 2026-11-13.

Bagging vs Boosting vs Stacking

How each method works

Bagging builds many independent trees on bootstrapped subsets of the data and averages their predictions. This reduces variance but retains the bias of each tree. In our case a RandomForestClassifier with 200 trees serves as the bagging baseline.

Boosting trains trees sequentially, each new tree focusing on the residual errors of the previous ensemble. Gradient Boosting thus reduces bias at the cost of higher variance and sensitivity to noise. We use GradientBoostingClassifier with 200 estimators.

Stacking layers a meta‑learner on top of several heterogeneous base models. The meta‑learner learns how to combine the base predictions, often capturing patterns that any single method misses. Here the base learners are the same Random Forest and Gradient Boosting models; a simple LogisticRegression acts as the meta‑learner.

Bias‑variance trade‑offs on the Random User feature set

The feature set contains a mix of numeric (age, registration age) and high‑cardinality categorical variables (city, state). Bagging handles high cardinality well because each tree can split on different categories, but it may miss subtle interactions between age and nationality. Boosting can model those interactions but may over‑fit the limited 500‑row sample. Stacking lets the meta‑learner decide when to trust the bagged prediction versus the boosted one, often yielding a lower combined bias while keeping variance in check.

Code snippets

Below are the core model constructors used in the full script.

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, StackingClassifier
from sklearn.linear_model import LogisticRegression

# Bagging
bagging = RandomForestClassifier(
    n_estimators=200,
    random_state=42,
    n_jobs=-1
)

# Boosting
boosting = GradientBoostingClassifier(
    n_estimators=200,
    learning_rate=0.1,
    random_state=42
)

# Stacking
stacking = StackingClassifier(
    estimators=[('rf', bagging), ('gb', boosting)],
    final_estimator=LogisticRegression(),
    passthrough=True,
    n_jobs=-1
)

Implementation Walkthrough

1. Data ingestion and flattening

We request the JSON payload, enforce a 10‑second timeout, and abort with a clear error if the network is slow or the response format changes.

2. Feature engineering

Numeric fields are passed through; categorical fields (nat, city, state) are one‑hot encoded with ColumnTransformer. The target (gender) is label‑encoded to 0/1.

3. Train‑test split

We use an 80/20 split with stratify=y to preserve the gender distribution and set random_state=42 for reproducibility.

4. Timing and evaluation

Each pipeline is fitted inside a time.time() block; accuracy is measured on the held‑out test set with accuracy_score. The script prints both metrics for direct comparison.

5. Failure handling

The script aborts with a helpful message if:

  • The HTTP request times out (network latency).
  • The JSON payload lacks the expected results key or any required field (schema change).

Limitations and Tradeoffs

While stacking delivered the highest accuracy in our experiment, it introduces several practical concerns:

  • Training time: Stacking trains each base learner plus the meta‑learner, roughly doubling the wall‑clock time compared to a single Random Forest.
  • Model complexity: More components mean a larger memory footprint and more dependencies to monitor in production.
  • Overfitting risk: With only 500 rows, a powerful meta‑learner can memorize the validation folds. Using a simple linear meta‑learner and limiting the number of base estimators mitigates this.
  • Feature explosion: One‑hot encoding high‑cardinality cities can blow up the feature matrix, increasing training time for all ensembles. Dimensionality reduction or target encoding are alternatives.
  • Latency: Real‑time inference must pass the input through the same preprocessing and three models, which may exceed strict latency budgets.

If any of these constraints dominate your use case, a well‑tuned Random Forest may be the more pragmatic choice.

Frequently Asked Questions

What size of dataset justifies stacking?

Stacking shines when you have enough data for the meta‑learner to generalize—typically a few thousand rows. Below 300 rows the variance introduced by stacking often outweighs the accuracy gain.

Can I replace the LogisticRegression meta‑learner with a more powerful model?

Yes, but doing so increases overfitting risk. Non‑linear meta‑learners (e.g., XGBoost) can capture complex relationships but should be paired with nested cross‑validation to avoid leakage.

How does class imbalance affect these ensembles?

All three methods are sensitive to imbalance because they optimize overall accuracy. Consider using class weights (e.g., class_weight='balanced') or evaluation metrics such as ROC‑AUC when the target distribution is skewed.

Is stacking compatible with online learning?

Standard scikit‑learn stacking is batch‑oriented. For streaming scenarios you would need incremental base learners (e.g., sklearn.linear_model.SGDClassifier) and a custom meta‑learning strategy.

Do I need to tune hyper‑parameters for each base learner?

Hyper‑parameter tuning improves every ensemble, but the relative ranking (bagging <  boosting <  stacking) tends to hold. A modest grid or random search on the Random Forest and Gradient Boosting usually suffices before stacking.

Recommendation and Next Step

For modest tabular problems like the Random User classification task, start with a well‑tuned Random Forest. If you observe a plateau in accuracy and have the engineering bandwidth to monitor additional latency, move to a stacking pipeline. The next logical step is to replace the manual random_state splits with a cross‑validation‑based hyper‑parameter search (e.g., Optuna) that optimizes both accuracy and training time for the stacked model.

Complete Script

#!/usr/bin/env python3
"""
Bagging, Boosting, and Stacking comparison on Random User data.

- Fetches 500 user records from the Random User API.
- Builds three ensembles: RandomForest (bagging), GradientBoosting (boosting),
  and Stacking (RF + GB + LogisticRegression).
- Prints accuracy and training time for each model.
"""

import sys
import time
import requests
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder, LabelEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score


def fetch_data(url: str, timeout: int = 10):
    """Fetch JSON data with timeout and basic error handling."""
    try:
        response = requests.get(url, timeout=timeout)
        response.raise_for_status()
    except requests.exceptions.Timeout:
        sys.exit(f"Error: Request timed out after {timeout} seconds.")
    except requests.exceptions.HTTPError as e:
        sys.exit(f"Error: HTTP request failed - {e}")
    data = response.json()
    if "results" not in data:
        sys.exit("Error: Unexpected response structure - 'results' key missing.")
    return data["results"]


def extract_features(records):
    """Extract features and target from raw records."""
    genders = []
    ages = []
    reg_ages = []
    nats = []
    cities = []
    states = []
    for rec in records:
        try:
            genders.append(rec["gender"])
            ages.append(rec["dob"]["age"])
            reg_ages.append(rec["registered"]["age"])
            nats.append(rec["nat"])
            cities.append(rec["location"]["city"])
            states.append(rec["location"]["state"])
        except KeyError as e:
            sys.exit(f"Error: Missing expected field {e} in a record.")
    X = np.array(
        list(zip(ages, reg_ages, nats, cities, states)),
        dtype=object
    )
    y = np.array(genders)
    return X, y


def build_preprocessor():
    """Create a ColumnTransformer that one-hot encodes categoricals."""
    categorical_idx = [2, 3, 4]
    numeric_idx = [0, 1]
    return ColumnTransformer(
        transformers=[
            ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_idx),
            ("num", "passthrough", numeric_idx),
        ]
    )


def train_models(X_train, X_test, y_train, y_test):
    """Train bagging, boosting, and stacking pipelines and return metrics."""
    results = {}

    bagging_pipe = Pipeline(
        steps=[
            ("preprocess", build_preprocessor()),
            ("clf", RandomForestClassifier(
                n_estimators=200,
                random_state=42,
                n_jobs=-1
            )),
        ]
    )
    start = time.time()
    bagging_pipe.fit(X_train, y_train)
    bag_time = time.time() - start
    bag_pred = bagging_pipe.predict(X_test)
    bag_acc = accuracy_score(y_test, bag_pred)
    results["Bagging"] = (bag_acc, bag_time)

    boosting_pipe = Pipeline(
        steps=[
            ("preprocess", build_preprocessor()),
            ("clf", GradientBoostingClassifier(
                n_estimators=200,
                learning_rate=0.1,
                random_state=42
            )),
        ]
    )
    start = time.time()
    boosting_pipe.fit(X_train, y_train)
    boost_time = time.time() - start
    boost_pred = boosting_pipe.predict(X_test)
    boost_acc = accuracy_score(y_test, boost_pred)
    results["Boosting"] = (boost_acc, boost_time)

    base_estimators = [
        ("rf", RandomForestClassifier(
            n_estimators=200,
            random_state=42,
            n_jobs=-1
        )),
        ("gb", GradientBoostingClassifier(
            n_estimators=200,
            learning_rate=0.1,
            random_state=42
        )),
    ]
    stacking_clf = StackingClassifier(
        estimators=base_estimators,
        final_estimator=LogisticRegression(),
        passthrough=True,
        n_jobs=-1,
    )
    stacking_pipe = Pipeline(
        steps=[
            ("preprocess", build_preprocessor()),
            ("clf", stacking_clf),
        ]
    )
    start = time.time()
    stacking_pipe.fit(X_train, y_train)
    stack_time = time.time() - start
    stack_pred = stacking_pipe.predict(X_test)
    stack_acc = accuracy_score(y_test, stack_pred)
    results["Stacking"] = (stack_acc, stack_time)

    return results


def main():
    api_url = "https://randomuser.me/api/?results=500&nat=us,gb,ca,au"
    raw_records = fetch_data(api_url)
    X, y_raw = extract_features(raw_records)

    le = LabelEncoder()
    y = le.fit_transform(y_raw)

    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=42, stratify=y
    )

    metrics = train_models(X_train, X_test, y_train, y_test)

    for name, (acc, t) in metrics.items():
        print(f"{name} accuracy: {acc:.4f}")
        print(f"{name} training time (s): {t:.2f}")


if __name__ == "__main__":
    main()

إرسال تعليق

Hi! How can we help you? Send us a message and we'll get back to you.