You’ve meticulously designed an A/B test, carefully defined your metrics, and launched it on your platform, expecting clean, actionable results. But what if the very foundation of your experiment—the user assignment—is subtly broken, leading you to fundamentally wrong conclusions? This isn't a hypothetical scenario; it's a production nightmare I've seen play out. Even sophisticated experimentation platforms can harbor insidious bugs that skew experiment populations, a phenomenon known as Sample Ratio Mismatch (SRM). This post is for data scientists and engineers who manage production experimentation systems and need to move beyond basic test setup to ensure the integrity of their experiment data, particularly when dealing with external data sources or complex user attributes. We'll build a robust detection and diagnosis mechanism to identify and pinpoint these elusive assignment errors before they invalidate your hard-won insights.
Key Takeaways
- SRM detection requires statistical rigor, typically using a Chi-Squared test, to identify significant deviations from expected group ratios.
- A proactive SRM monitoring system is essential for production A/B tests to catch assignment bugs early, preventing erroneous product decisions.
- Diagnosing SRM goes beyond overall detection; granular breakdown by user attributes (e.g., country, gender) is critical to pinpoint the root cause.
- Subtle bugs in user assignment logic, especially those tied to specific user segments or external data, are common and can invalidate entire experiments.
- Robust error handling and data validation are non-negotiable when fetching and processing external user data for experimentation.
The Problem: The Invisible Bias
Imagine you're running an A/B test to evaluate a new feature rollout. Your platform reports a 50/50 split between control and treatment groups, and initial metrics look promising for the new feature. You launch it, only to find the real-world impact doesn't match the test results. What happened? Often, the culprit is a subtle SRM. Perhaps a new caching layer misrouted users from a specific country, or a backend service bugged out when processing a particular user attribute, inadvertently dropping them from the experiment or assigning them disproportionately. These aren't obvious crashes; they're silent killers of statistical validity, making your A/B test results meaningless. My goal here is to show you how to not just detect these silent killers, but to effectively diagnose them.
Data and Sources
For this exploration, we'll simulate user traffic using the Random User API. This API provides realistic, albeit synthetic, user profiles that allow us to simulate diverse user attributes like gender and country, which are crucial for engineering and diagnosing subtle SRM. We'll be using standard Python libraries: `requests` for API interaction, `hashlib` for consistent user assignment, `pandas` for data structuring, `scipy.stats` for statistical testing, and `matplotlib.pyplot` along with `seaborn` for visualization.
Data accessed on 2024-07-29.
Step 1 — Simulating User Traffic and Baseline Assignment
The first sub-problem is establishing a controlled environment: generating diverse "user" data and simulating a *correct*, balanced A/B group assignment. This baseline is what we'll later corrupt to demonstrate SRM. We need a way to fetch a reasonable number of users and then assign them deterministically to either Group A (control) or Group B (treatment) with an equal probability.
I start by fetching a batch of users from the Random User API. Then, for consistent assignment, I use a simple hashing technique. Each user gets a unique identifier, and hashing that ID allows for a deterministic, repeatable assignment to a group. This ensures that the same user always lands in the same group, which is critical for real-world A/B testing.
import requests
import hashlib
import json
import pandas as pd
from scipy.stats import chi2_contingency
import matplotlib.pyplot as plt
import seaborn as sns
import collections
# --- Step 1: Simulating User Traffic and Baseline Assignment ---
def fetch_users(num_users: int) -> list:
"""Fetches a specified number of random users from the API."""
users = []
try:
for _ in range(num_users // 50): # API returns max 50 per request
response = requests.get('https://randomuser.me/api/?results=50')
response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)
users.extend(response.json()['results'])
# Fetch remaining users if num_users is not a multiple of 50
if num_users % 50 > 0:
response = requests.get(f'https://randomuser.me/api/?results={num_users % 50}')
response.raise_for_status()
users.extend(response.json()['results'])
except requests.exceptions.RequestException as e:
print(f"Error fetching users: {e}")
return []
except json.JSONDecodeError as e:
print(f"Error decoding JSON response: {e}")
return []
return users
def parse_user_data(user_raw: dict) -> dict:
"""Parses relevant attributes from a raw user dictionary."""
try:
return {
'user_id': user_raw['login']['uuid'],
'gender': user_raw['gender'],
'country': user_raw['location']['country'],
'state': user_raw['location']['state'],
'city': user_raw['location']['city']
}
except KeyError as e:
# Handle cases where expected keys might be missing
print(f"Warning: Missing key in user data: {e} for user {user_raw.get('login', {}).get('uuid', 'unknown')}")
return None # Indicate a parsing failure
def consistent_hash_assignment(user_id: str, num_groups: int = 2) -> str:
"""Assigns a user to a group (A or B) based on a consistent hash."""
# Using SHA256 for a robust hash, then modulo for group assignment
hash_object = hashlib.sha256(user_id.encode())
hash_int = int(hash_object.hexdigest(), 16)
group_index = hash_int % num_groups
return 'A' if group_index == 0 else 'B'
# Example usage (not part of final script, just for illustration)
# raw_users = fetch_users(100)
# processed_users = [parse_user_data(u) for u in raw_users if parse_user_data(u) is not None]
# df_baseline = pd.DataFrame(processed_users)
# df_baseline['group'] = df_baseline['user_id'].apply(consistent_hash_assignment)
The `fetch_users` function handles fetching batches of users and includes basic `try/except` for network and JSON decoding errors. `parse_user_data` extracts specific attributes, crucial for later diagnosis, and also includes error handling for missing keys. Finally, `consistent_hash_assignment` takes a user ID, hashes it, and assigns the user to 'A' or 'B' based on the hash's parity, ensuring an expected 50/50 split.
Step 2 — Engineering the Subtle SRM Bug (The Production Nightmare)
Now for the fun part: introducing a realistic, hard-to-spot bug into the assignment logic that causes SRM based on specific user attributes. This mimics real-world deployment errors where, for instance, a new feature might only be partially rolled out, or a filter condition is subtly wrong. I've designed `faulty_assignment` to target users from 'Nepal' or 'female' users, giving them a higher probability of being assigned to Group B or even dropping them from assignment with a small probability. This kind of selective mis-assignment is incredibly difficult to spot without explicit checks.
# --- Step 2: Engineering the Subtle SRM Bug ---
def faulty_assignment(user_data: dict) -> str | None:
"""
Introduces a subtle SRM bug:
- Users from 'Nepal' have a 70% chance of going to B, 30% to A.
- Female users (not from Nepal) have a 60% chance of going to B, 40% to A.
- A small percentage (5%) of users from 'Nepal' are dropped (None).
- Other users are assigned 50/50.
"""
user_id = user_data['user_id']
country = user_data['country']
gender = user_data['gender']
# Simulate a small percentage of users being dropped due to a bug
if country == 'Nepal' and hashlib.sha256(user_id.encode()).hexdigest()[-2:] < '0D': # ~5% chance
return None # User dropped from experiment
# Bias users from 'Nepal' towards Group B
if
