Working with real-world data is rarely as clean as the examples in textbooks. I've spent countless hours wrestling with datasets from various APIs and internal systems, where missing values lurk in unexpected columns, and outliers skew critical metrics. It's a common pain point for data scientists and engineers: you have a promising idea, but the raw data is a tangled mess, threatening to derail your entire analysis or machine learning model before you even start. This post will walk you through my pragmatic approach to tackling these challenges, using a practical example from the Random User API. By the end, you'll have a clear understanding of how to identify, handle, and prepare your own messy data, turning raw information into a clean, usable foundation for robust insights.
Key Takeaways
- Identifying missing values with pandas and visualizing their patterns is the first critical step.
- Handling missing values through imputation (mean, median, mode) requires understanding the data distribution and the impact of each method.
- Detecting outliers using statistical methods like IQR and Z-score helps prevent skewed analyses and models.
- Visualizing data before and after cleaning provides crucial insights into the effectiveness of your techniques.
- Scaling numerical features using StandardScaler or MinMaxScaler is essential for many machine learning algorithms and improves model performance.
The Problem
Imagine you're building a system that processes user profiles, perhaps for a recommendation engine or a demographic analysis. You pull data from an external API, expecting a pristine, uniform stream. What you often get instead is a mix of complete records, partially empty fields, and some values that are statistically improbable. These inconsistencies aren't just cosmetic; they can lead to biased models, incorrect statistical inferences, and ultimately, poor business decisions. My goal here is to demonstrate how to systematically address these issues, ensuring your data is not just present, but also reliable and fit for purpose.
Data and Sources
For this walkthrough, we'll be fetching data directly from the Random User API, a publicly available service that generates random user data. This API is excellent for simulating real-world data collection, as its output can sometimes feature inconsistencies or nested structures that mimic production challenges.
- Random User API
- pandas Documentation
- NumPy Documentation
- Matplotlib Documentation
- Seaborn Documentation
- scikit-learn StandardScaler
- scikit-learn MinMaxScaler
Data accessed on 2024-07-20.
Step 1 — Data Ingestion
The first hurdle is always getting the data into a usable format. The Random User API returns JSON, often with deeply nested structures. Simply calling pd.DataFrame() on the raw JSON won't cut it; you'll end up with columns containing dictionaries, not flat values. The sub-problem here is transforming this nested JSON into a flat, tabular pandas DataFrame suitable for analysis.
I use requests to fetch the data and then json_normalize from pandas to flatten the nested JSON. This function is a lifesaver for handling semi-structured data, automatically expanding nested dictionaries into separate columns with a sensible naming convention.
import requests
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.impute import SimpleImputer
from scipy.stats import zscore
def fetch_user_data(num_users=100):
"""Fetches random user data from the Random User API."""
url = f"https://randomuser.me/api/?results={num_users}"
try:
response = requests.get(url, timeout=10)
response.raise_for_status() # Raise an exception for HTTP errors
data = response.json()
return data['results']
except requests.exceptions.RequestException as e:
print(f"Error fetching data: {e}")
return None
except ValueError as e: # Catch JSON decoding errors
print(f"Error decoding JSON: {e}")
return None
def create_dataframe(raw_data):
"""Flattens raw JSON data into a pandas DataFrame."""
if not raw_data:
return pd.DataFrame()
# Flatten the JSON data
df = pd.json_normalize(raw_data)
# Introduce some artificial missing data for demonstration purposes
# as the API often returns very clean data for common fields.
# For example, let's make 5% of 'email' values missing.
missing_indices_email = np.random.choice(df.index, size=int(len(df) * 0.05), replace=False)
df.loc[missing_indices_email, 'email'] = np.nan
# Introduce missing data in 'location.postcode'
missing_indices_postcode = np.random.choice(df.index, size=int(len(df) * 0.1), replace=False)
df.loc[missing_indices_postcode, 'location.postcode'] = np.nan
# Introduce some non-standard values in 'gender' for later cleaning demonstration
non_standard_indices_gender = np.random.choice(df.index, size=int(len(df) * 0.03), replace=False)
df.loc[non_standard_indices_gender, 'gender'] = 'N/A'
return df
The `fetch_user_data` function handles the API call and basic error checking. The `create_dataframe` function then takes the list of user dictionaries and leverages `pd.json_normalize` to expand nested fields like name.first or location.city into distinct columns. I've also explicitly introduced some synthetic missing values and non-standard entries to ensure we have real problems to solve later, as the Random User API can sometimes be surprisingly clean for its primary fields.
Step 2 — Identifying Missing Values
Once the data is in a DataFrame, the next challenge is to understand where the gaps are. The sub-problem here is systematically identifying all missing values and visualizing their distribution to uncover patterns that might inform our imputation strategy.
I typically start with df.isnull().sum() to get a quick count of missing values per column. For a more intuitive understanding, especially with larger datasets, a heatmap using Seaborn is invaluable. It visually highlights columns or rows with high missingness, making patterns immediately obvious.
def identify_missing_values(df):
"""Identifies and visualizes missing values."""
print("--- Missing Values Summary ---")
missing_counts = df.isnull().sum()
missing_percentage = (df.isnull().sum() / len(df)) * 100
missing_info = pd.DataFrame({
'Missing Count': missing_counts,
'Missing Percentage': missing_percentage
})
print(missing_info[missing_info['Missing Count'] > 0].sort_values(by='Missing Count', ascending=False))
# Visualize missing values
plt.figure(figsize=(12, 6))
sns.heatmap(df.isnull(), cbar=False, cmap='viridis')
plt.title('Missing Values Heatmap')
plt.tight_layout()
plt.savefig('missing_values_heatmap.png')
plt.close()
print("\nMissing values heatmap saved to 'missing_values_heatmap.png'.")
The output from identify_missing_values provides both a numerical count and percentage, giving you precise figures. The heatmap offers a macro view, allowing you to quickly spot if missingness is clustered in certain columns or rows, which could indicate data collection issues or specific data types that are often optional.
Step 3 — Handling Missing Values
With missing values identified, the next step is to decide how to handle them. The sub-problem is choosing an appropriate imputation strategy that minimizes bias and preserves data integrity. This is often where mean, median, or mode imputation come into play.
For numerical columns, mean imputation replaces missing values with the average, while median imputation uses the middle value. Median is generally more robust to outliers. For categorical columns, mode imputation (most frequent value) is a common choice. I often use SimpleImputer from scikit-learn for a more robust and pipeline-friendly approach, though direct pandas fillna works for quick scripts.
def handle_missing_values(df):
"""Handles missing values using imputation techniques
