How to Catalog a Simple JSON API in OpenMetadata and Apache Atlas

You have already hardened a Todo‑API pipeline with Netflix‑inspired patterns; the next step is to make that API visible to your enterprise catalog so downstream analysts can discover, trust, and lineage‑track the data. This post shows, with a single 45‑line Python script, how to turn the public JSONPlaceholder Todo API into catalog entities in both OpenMetadata and Apache Atlas, and it explains the practical trade‑offs you’ll face when choosing one system over the other.

Key Takeaways

  • Both OpenMetadata and Atlas can be fed from a tiny Python ingest layer; the code difference is limited to payload shape and endpoint URL.
  • OpenMetadata gives richer UI‑driven lineage (automatic job‑run links) but requires a running metadata service; Atlas offers tighter Hadoop integration at the cost of more manual lineage stitching.
  • Operational overhead scales with the number of distinct entity types you expose – a simple Todo table stays lightweight in either catalog, but adding complex transformations quickly favors OpenMetadata’s model‑driven approach.

Data and Sources

The example uses the stable JSONPlaceholder Todos endpoint, which returns exactly 200 records with the fields id, userId, title, and completed. The source was reviewed on 2026-11-15.

OpenMetadata vs Apache Atlas: Practical Comparison

AspectOpenMetadataApache Atlas
API modelREST / JSON with explicit Table, Column, Lineage objectsREST / JSON but relies on entity type definitions (e.g., hive_table)
UI lineage depthAuto‑generated job‑run nodes, end‑to‑end graph viewManual edge creation; UI shows edges but no job context
Setup complexityDocker‑compose or Helm chart; service must be reachableOften bundled with Hadoop/YARN; requires Kerberos or LDAP for production
Schema evolution supportVersioned schema objects, diff UISchema changes are just new attributes; diff tooling is external
Operational overheadSingle metadata service; health checks, backupsMultiple services (atlas, solr, hbase); higher ops burden

Mapping the Todo API to Catalog Entities

Common entity model

Both catalogs need a database container, a table representing the Todo list, and column definitions for each field. The mapping is straightforward:

  • database: jsonplaceholder
  • table: todos
  • Columns: id (int), userId (int), title (string), completed (boolean)

OpenMetadata payload shape

{
  "name": "todos",
  "database": "jsonplaceholder",
  "columns": [
    {"name": "id", "dataType": "INT"},
    {"name": "userId", "dataType": "INT"},
    {"name": "title", "dataType": "STRING"},
    {"name": "completed", "dataType": "BOOLEAN"}
  ]
}

Apache Atlas payload shape

{
  "entity": {
    "typeName": "hive_table",
    "attributes": {
      "qualifiedName": "jsonplaceholder.todos@primary",
      "name": "todos",
      "owner": "catalog_ingest",
      "columns": [
        {"typeName": "hive_column", "attributes": {"name": "id", "type": "int"}},
        {"typeName": "hive_column", "attributes": {"name": "userId", "type": "int"}},
        {"typeName": "hive_column", "attributes": {"name": "title", "type": "string"}},
        {"typeName": "hive_column", "attributes": {"name": "completed", "type": "boolean"}}
      ]
    }
  }
}

Ingest Script Walkthrough

The script follows three deterministic steps:

  1. Fetch the Todo list with a 5‑second timeout; raise a clear error if the HTTP status is not 200.
  2. Validate that the response is a list of 200 objects and that each object contains the four required keys. Missing keys raise a ValueError that includes the offending record index.
  3. Generate two JSON payloads – one for OpenMetadata, one for Atlas – compute a SHA‑256 checksum for each, and print both the payload and its hash. No external catalog service is called, keeping the example zero‑setup.

Two realistic failure cases are covered:

  • Network timeout – caught as requests.exceptions.Timeout with a user‑friendly message.
  • Schema mismatch – caught as ValueError with details about which field is absent.

Limitations and Tradeoffs

  • Lineage granularity: OpenMetadata can attach a lineage edge directly to the API fetch job, whereas Atlas would require a separate process to emit a Process entity.
  • Operational footprint: Running OpenMetadata locally adds a single Docker container; Atlas typically needs a full stack (Solr, HBase), which may be overkill for a small API.
  • UI richness: OpenMetadata’s UI displays column‑level tags and owners out‑of‑the‑box; Atlas UI shows attributes but lacks built‑in tagging UI, meaning you must manage tags via API calls.
  • Future scaling: If you later add incremental loads or join the Todo data with other sources, OpenMetadata’s model‑driven lineage will scale more cleanly, while Atlas will require manual edge updates.

Frequently Asked Questions

Do I need to run both catalogs in production?

No. Choose the one that aligns with your existing stack: OpenMetadata for cloud‑native, service‑oriented environments; Atlas for Hadoop‑centric ecosystems.

Can the script handle pagination or larger APIs?

The example assumes a single‑page response. For paginated APIs you would loop over next URLs, concatenate results, and then run the same validation logic.

How do I persist the payloads to the actual catalog?

Replace the print statements with an HTTP POST to /api/v1/tables (OpenMetadata) or /api/atlas/v2/entity (Atlas), using the same JSON bodies.

What if my API schema changes?

The validation step will raise a ValueError. You can catch it and trigger a CI alert, or you can version the catalog entities by appending a schema version suffix to the table name.

Is there a way to automate this in Airflow?

Yes. The script is idempotent; wrap it in a PythonOperator and set retries and execution_timeout to match the patterns described in the Netflix‑inspired post.

Recommendation and Next Step

If your organization already runs OpenMetadata for data‑mesh initiatives, adopt the OpenMetadata branch of the script and configure the POST endpoint to your metadata service. Hadoop‑centric teams should extend the Atlas branch, register the payload with the Atlas REST API, and consider adding Atlas lineage hooks for downstream Hive jobs. The practical next step is to embed the script in the Airflow DAG you built for the resilient Todo pipeline, schedule it daily, and monitor the checksum logs to ensure the catalog stays in sync.

Complete Script

#!/usr/bin/env python3
"""
Catalog the JSONPlaceholder Todo API in OpenMetadata and Apache Atlas.

The script:
  1. Fetches the Todo list (200 records) with a timeout.
  2. Validates required fields.
  3. Builds two payloads – one for OpenMetadata, one for Atlas.
  4. Prints each payload and a SHA‑256 checksum.
No external catalog services are invoked; the script is self‑contained.
"""

import json
import hashlib
import sys
from typing import List, Dict

import requests

API_URL = "https://jsonplaceholder.typicode.com/todos"
TIMEOUT = 5  # seconds
REQUIRED_FIELDS = {"id", "userId", "title", "completed"}


def fetch_todos() -> List[Dict]:
    """Fetch the Todo list with robust error handling."""
    try:
        response = requests.get(API_URL, timeout=TIMEOUT)
        response.raise_for_status()
    except requests.exceptions.Timeout:
        sys.exit("❌ Request timed out after %s seconds" % TIMEOUT)
    except requests.exceptions.HTTPError as e:
        sys.exit(f"❌ HTTP error: {e}")
    except requests.exceptions.RequestException as e:
        sys.exit(f"❌ Network error: {e}")

    data = response.json()
    if not isinstance(data, list):
        sys.exit("❌ Unexpected response shape: expected a list")
    if len(data) != 200:
        sys.exit(f"❌ Expected 200 records, got {len(data)}")
    return data


def validate_records(records: List[Dict]) -> None:
    """Ensure every record contains the required fields."""
    for idx, rec in enumerate(records):
        missing = REQUIRED_FIELDS - rec.keys()
        if missing:
            sys.exit(
                f"❌ Record {idx} is missing fields: {', '.join(sorted(missing))}"
            )


def build_openmetadata_payload() -> Dict:
    """Create the OpenMetadata JSON payload."""
    return {
        "name": "todos",
        "database": "jsonplaceholder",
        "columns": [
            {"name": "id", "dataType": "INT"},
            {"name": "userId", "dataType": "INT"},
            {"name": "title", "dataType": "STRING"},
            {"name": "completed", "dataType": "BOOLEAN"},
        ],
    }


def build_atlas_payload() -> Dict:
    """Create the Apache Atlas JSON payload."""
    return {
        "entity": {
            "typeName": "hive_table",
            "attributes": {
                "qualifiedName": "jsonplaceholder.todos@primary",
                "name": "todos",
                "owner": "catalog_ingest",
                "columns": [
                    {
                        "typeName": "hive_column",
                        "attributes": {"name": "id", "type": "int"},
                    },
                    {
                        "typeName": "hive_column",
                        "attributes": {"name": "userId", "type": "int"},
                    },
                    {
                        "typeName": "hive_column",
                        "attributes": {"name": "title", "type": "string"},
                    },
                    {
                        "typeName": "hive_column",
                        "attributes": {"name": "completed", "type": "boolean"},
                    },
                ],
            },
        }
    }


def sha256_hash(obj: Dict) -> str:
    """Return a SHA‑256 hash of a JSON‑serialised object."""
    json_bytes = json.dumps(obj, sort_keys=True).encode("utf-8")
    return hashlib.sha256(json_bytes).hexdigest()


def main() -> None:
    records = fetch_todos()
    validate_records(records)

    om_payload = build_openmetadata_payload()
    atlas_payload = build_atlas_payload()

    om_hash = sha256_hash(om_payload)
    atlas_hash = sha256_hash(atlas_payload)

    print(f"OpenMetadata payload (hash={om_hash}):")
    print(json.dumps(om_payload, indent=2, sort_keys=True))
    print()
    print(f"Atlas payload (hash={atlas_hash}):")
    print(json.dumps(atlas_payload, indent=2, sort_keys=True))
    print()
    print(f"✅ {len(records)} records processed, payloads generated")


if __name__ == "__main__":
    main()

Post a Comment

Hi! How can we help you? Send us a message and we'll get back to you.