Ensuring Data Integrity: Implementing Great Expectations for F1 Racing Data

The Problem: When Bad Data Threatens Your F1 Insights

Have you ever spent hours debugging a downstream system, only to discover the root cause was a subtle, silent degradation in your upstream data source? I certainly have. I was deep into building a dashboard for F1 race statistics, pulling data from various APIs to track everything from lap times to championship standings. Everything felt smooth, a real triumph after wrestling with streaming data pipelines, as we discussed in Mastering Streaming Data Processing with Kafka and Python. Then, one morning, my dashboard showed "N/A" for a recent race location, and country codes were suddenly malformed. Hours of debugging later, I traced it back to a subtle, undocumented change in one of my upstream data sources. The API hadn't failed, but the data quality had silently degraded.

This experience hammered home a critical lesson: ingesting data, even from seemingly reliable APIs like the Open F1 API, is only half the battle. Without robust data validation, you're constantly walking a tightrope, risking incorrect insights, broken dashboards, and wasted development time. For data engineers, especially those of us who've moved beyond basic streaming and are now dealing with the complexities of production pipelines, ensuring data accuracy and reliability is paramount. That's precisely why I leaned into Great Expectations – a powerful tool that lets you define, enforce, and visualize data quality expectations, offering a proactive shield against data chaos.

Step 1: Setting Up Our Data Playground and Great Expectations

The first hurdle is always getting your tools and data ready. For this project, I wanted to validate F1 race meeting data from the Open F1 API. This meant making an HTTP request and then preparing that JSON response for Great Expectations. While Great Expectations can connect to various data sources, for API data like this, it's often easiest to load it into a Pandas DataFrame first.

Here’s how I started, fetching the 2024 F1 meeting data and converting it into a DataFrame:

import requests
import pandas as pd
from great_expectations.data_context import DataContext
from great_expectations.core import ExpectationConfiguration
from great_expectations.exceptions import InvalidExpectationConfigurationError, GreatExpectationsError
import os

def load_f1_data(year: int = 2024) -> pd.DataFrame:
    """Fetches F1 meeting data from Open F1 API and returns a DataFrame."""
    api_url = f"https://api.openf1.org/v1/meetings?year={year}"
    try:
        response = requests.get(api_url, timeout=10)
        response.raise_for_status() # Raises HTTPError for bad responses (4xx or 5xx)
        raw_data = response.json()
        if not raw_data:
            print(f"Warning: No data received from API for year {year}.")
            return pd.DataFrame()
        return pd.DataFrame(raw_data)
    except requests.exceptions.Timeout:
        print(f"Error: API request timed out after 10 seconds for {api_url}")
        return pd.DataFrame()
    except requests.exceptions.RequestException as e:
        print(f"Error fetching data from Open F1 API: {e}")
        return pd.DataFrame()

I wrapped the API call in a function with `try-except` blocks to handle common network issues or API errors, which is crucial for any real-world data ingestion pipeline. Next, I needed to initialize Great Expectations. For programmatic use, I often skip the `great_expectations init` CLI command and directly instantiate a `DataContext` in a temporary directory. This keeps my project clean and self-contained for testing or one-off validations.

Step 2: Crafting Expectations for F1 Meeting Data

With the data loaded, the next challenge was defining what "good" F1 data looks like. This is where Great Expectations shines. You define "Expectations" – assertions about your data – that are then checked against your dataset. For F1 meeting data, I wanted to ensure essential columns exist, their values are of the correct type, and critical fields like `meeting_name`, `location`, and `country_code` are never null.

Here are some of the expectations I defined for our F1 meetings DataFrame:

# Inside the main validation logic
batch_request = {
    "datasource_name": "f1_datasource",
    "data_connector_name": "default_inferred_data_connector_name",
    "data_asset_name": "f1_meetings",
    "batch_spec_passthrough": {"batch_data": df},
}

# Define our expectations
expectation_suite_name = "f1_meetings_suite"
context.create_expectation_suite(expectation_suite_name=expectation_suite_name, overwrite_existing=True)

# Expectation 1: Required columns exist
context.add_expectation(
    expectation_suite_name=expectation_suite_name,
    expectation_configuration=ExpectationConfiguration(
        expectation_type="expect_column_to_exist",
        kwargs={"column": "meeting_key"}
    )
)
context.add_expectation(
    expectation_suite_name=expectation_suite_name,
    expectation_configuration=ExpectationConfiguration(
        expectation_type="expect_column_to_exist",
        kwargs={"column": "meeting_name"}
    )
)
# ... more column existence expectations ...

# Expectation 2: No null values in critical columns
context.add_expectation(
    expectation_suite_name=expectation_suite_name,
    expectation_configuration=ExpectationConfiguration(
        expectation_type="expect_column_values_to_not_be_null",
        kwargs={"column": "meeting_name"}
    )
)
context.add_expectation(
    expectation_suite_name=expectation_suite_name,
    expectation_configuration=ExpectationConfiguration(
        expectation_type="expect_column_values_to_not_be_null",
        kwargs={"column": "location"}
    )
)

# Expectation 3: Correct data types
context.add_expectation(
    expectation_suite_name=expectation_suite_name,
    expectation_configuration=ExpectationConfiguration(
        expectation_type="expect_column_values_to_be_of_type",
        kwargs={"column": "meeting_key", "type_": "int"}
    )
)
context.add_expectation(
    expectation_suite_name=expectation_suite_name,
    expectation_configuration=ExpectationConfiguration(
        expectation_type="expect_column_values_to_be_of_type",
        kwargs={"column": "meeting_name", "type_": "str"}
    )
)

# Expectation 4: Country codes adhere to a specific format (e.g., 3 uppercase letters)
context.add_expectation(
    expectation_suite_name=expectation_suite_name,
    expectation_configuration=ExpectationConfiguration(
        expectation_type="expect_column_values_to_match_regex",
        kwargs={"column": "country_code", "regex": r"^[A