Picture this: you've meticulously built a data pipeline, perhaps even using Pydantic for robust schema validation on incoming API data, just as we explored in an earlier post. Your data flows smoothly into your Data Vault 2.0 structures, feeding MLflow-tracked models. Everything seems perfect—until a critical report suddenly shows nonsensical values, or your model's performance quietly degrades. The schema of your external API data hasn't changed, so what went wrong? The culprit, more often than not, is a subtle drift in the *content* or *distribution* of the API data itself. This is where basic schema validation falls short. This post will guide you through implementing Great Expectations to proactively define, validate, and monitor the nuanced quality of external API data, allowing you to catch these insidious issues before they corrupt your downstream systems and ensure the integrity of your entire data ecosystem.
Key Takeaways
- API data quality extends far beyond basic schema validation; content and distribution shifts can silently break downstream systems.
- Great Expectations provides a robust framework for defining explicit data quality expectations, acting as a "data contract" for external sources.
- Leverage custom expectations and dynamic validation to catch subtle anomalies that fixed schema checks would miss.
- Integrate Great Expectations checkpoints into your data pipelines for automated, auditable data quality validation.
- Proactive data quality testing saves significant engineering effort by preventing data corruption and debugging complex downstream failures.
The Unseen Menace: Why API Data Quality Goes Beyond Schema Validation
We've all been there: an external API, a critical data source for our analytics or machine learning models, starts sending data that looks "correct" on the surface. The JSON structure matches your Pydantic models perfectly. No validation errors. Yet, the numbers in your reports are off, or your recommendation engine starts suggesting irrelevant items. This isn't a schema violation; it's a *data quality* issue. Perhaps a `status` field that used to only contain "active" or "inactive" now includes "pending" with no prior notice. Or a `price` field, while still numeric, suddenly sees its average value drop by 50% due to an upstream system error. These are the kinds of silent killers that Great Expectations is designed to catch, ensuring your data pipelines are not just structurally sound, but also semantically robust.
Data and Sources
For this tutorial, we'll be working with a common pattern: fetching data from a public REST API. Our source for "posts" data will be JSONPlaceholder, a free fake API for testing and prototyping.
- JSONPlaceholder Posts API: https://jsonplaceholder.typicode.com/posts
- Great Expectations Documentation: https://docs.greatexpectations.io/docs/
Data accessed on 2024-07-29.
Step 1 — Laying the Foundation: Initializing a Great Expectations Data Context and Datasource
Before we can define any expectations, Great Expectations needs a "Data Context" to store configurations, expectations, and validation results. Think of it as the central nervous system for your data quality efforts. We'll initialize this context and then connect it to our external API data source. The key here is setting up a "Pandas Datasource" since we'll be loading our JSON into a Pandas DataFrame for validation.
import os
import requests
import pandas as pd
from great_expectations.data_context import DataContext
from great_expectations.data_context.types.base import DataContextConfig, DatasourceConfig, PandasDatasourceConfig
def initialize_gx_context(project_root_dir="gx"):
# Ensure the directory exists
os.makedirs(project_root_dir, exist_ok=True)
os.chdir(project_root_dir)
# Initialize a new DataContext if one doesn't exist
if not os.path.exists("great_expectations"):
context = DataContext.create(project_root_dir=os.getcwd())
else:
context = DataContext(project_root_dir=os.getcwd())
# Add a Pandas Datasource for our API data
datasource_name = "jsonplaceholder_api_datasource"
if datasource_name not in context.list_datasources():
datasource_config = PandasDatasourceConfig(
name=datasource_name,
module_name="great_expectations.datasource",
class_name="PandasDatasource"
)
context.add_datasource(datasource_config=datasource_config)
os.chdir("..") # Change back to original directory
return context
# This snippet initializes the GX context
# context = initialize_gx_context()
# print(f"Great Expectations context initialized at: {context.root_directory}")
The `initialize_gx_context` function first ensures a project directory exists and then creates or loads a Great Expectations Data Context. Crucially, it then adds a `PandasDatasource`. While Great Expectations supports many data connectors, for API data that we'll load into memory, the Pandas Datasource is ideal as it allows us to validate DataFrames directly.
Step 2 — Crafting Robust Expectations: Beyond Basic Schema Checks
With our context ready, the next step is to define the expectations. This is where we move beyond simple schema validation. We'll create an "Expectation Suite" — a collection of assertions about our data. For the JSONPlaceholder posts, we'll define expectations that ensure not just the presence and type of columns, but also their content and distribution characteristics. This includes checking for unique IDs, valid ranges for user IDs, specific types of titles, and even the expected length of the `body` content.
from great_expectations.dataset import PandasDataset
def create_expectation_suite(context: DataContext, data: pd.DataFrame, suite_name: str):
batch = context.get_batch(
batch_kwargs={
"datasource": "jsonplaceholder_api_datasource",
"batch_data": data,
"data_asset_name": suite_name # Use suite_name as data_asset_name for simplicity
},
expectation_suite_name=suite_name
)
# Basic schema checks (still important!)
batch.expect_column_to_exist("userId")
batch.expect_column_to_exist("id")
batch.expect_column_to_exist("title")
batch.expect_column_to_exist("body")
batch.expect_column_values_to_be_of_type("userId", "int")
batch.expect_column_values_to_be_of_type("id", "int")
batch.expect_column_values_to_be_of_type("title", "str")
batch.expect_column_values_to_be_of_type("body", "str")
# Beyond schema: Content and Distribution Expectations
batch.expect_column_values_to_be_unique("id")
batch.expect_column_values_to_not_be_null("id")
batch.expect_column_values_to_be_between("userId", min_value=1, max_value=10) # userId should be between 1 and 10
batch.expect_column_value_lengths_to_be_between("title", min_value=10, max_value=100) # Title length check
batch.expect_column_value_lengths_to_be_between("body", min_value=50, max_value=500) # Body content length
# Example of a more advanced expectation: checking for unexpected values in title
# (e.g., if titles suddenly contain "SPAM" or "ADVERTISEMENT")
batch.expect_column_values_to_not_match_regex("title", r"(?i)(spam|advertisement)")
batch.save_expectation_suite(discard_failed_expectations=False)
print(f"Expectation suite '{suite_name}' created and saved.")
return suite_name
# This snippet creates the expectation suite
# suite_name = create_expectation_suite(context, df_posts, "posts_api_quality_suite")
Here, we're creating a `PandasDataset` from our DataFrame and then applying a series of `expect_` methods. Notice how we go beyond `expect_column_to_exist` and `expect_column_values_to_be_of_type`. We're asserting that `id` values are unique, `userId` falls within a specific range (1-10 for JSONPlaceholder), and even that `title` and `body` lengths are within reasonable bounds. The `expect_column_values_to_not_match_regex` on `title` is a powerful example of catching content drift – imagine if an external API started injecting marketing phrases into what should be neutral post titles.
Step 3 — Orchestrating Validation: Building a Production-Ready Checkpoint
Defining expectations is one thing; consistently running them and generating reports is another. Great Expectations "Checkpoints" are the mechanism for orchestrating validation runs. A checkpoint bundles one or more batches of data with one or more expectation suites and
