Traditional data catalogs are excellent at managing structured data, but they often fall short when dealing with external, unstructured sources like competitive intelligence feeds from engineering blogs. This limitation can lead to blind spots, hindering discoverability, lineage tracking, and governance. In this post, we'll explore a concrete, production-ready approach to bring these critical external data assets and their derived insights into your data catalog, ensuring your intelligence pipelines are fully auditable and understandable. We'll use the Slack Engineering RSS feed as a real-world example, demonstrating how to extend OpenMetadata with custom entities and relationships.
Key Takeaways
- Custom entity types and relationships can be used to catalog external, unstructured data sources like RSS feeds.
- OpenMetadata provides a flexible framework for extending data catalogs with custom metadata.
- Programmatic ingestion of RSS data into a data catalog can be achieved using Python and the OpenMetadata API.
The Problem
The lack of standardization in external data sources like RSS feeds makes it challenging to integrate them into traditional data catalogs. Out-of-the-box catalog connectors often fall short, and custom solutions are needed to bridge this gap. In this post, we'll address this problem by demonstrating how to define custom entity types and relationships for external feeds and then ingest the data into a data catalog.
Data and Sources
We'll be using the Slack Engineering RSS feed as our data source, which is available at https://slack.engineering/feed/. The feed provides information about recent posts on the Slack Engineering blog, including titles, links, and summaries. Data accessed on 2024-09-16.
Step 1 — The Gap: Why Standard Catalog Connectors Fall Short for Unstructured Feeds
This step highlights the limitations of out-of-the-box catalog connectors for non-database sources, demonstrating the need for custom types. We'll use the `feedparser` library to extract data from the Slack Engineering RSS feed.
import feedparser
feed = feedparser.parse('https://slack.engineering/feed/')
for entry in feed.entries[:5]:
print(entry.title, entry.link)
Step 2 — Architecting Custom Metadata: Defining `EngineeringBlogFeed` and `BlogArticle` Entities
This step addresses how to conceptualize and define new data asset types (custom entities) and their properties within OpenMetadata to represent external feeds and their individual articles. We'll define two custom entities: `EngineeringBlogFeed` and `BlogArticle`.
from openmetadata.entity import Entity
class EngineeringBlogFeed(Entity):
# define properties for the feed entity
pass
class BlogArticle(Entity):
# define properties for the article entity
pass
Step 3 — Programmatic Ingestion: Syncing RSS Data to Your Catalog with Idempotency
This step demonstrates how to connect to OpenMetadata, create/update the custom entity types, and then ingest individual RSS entries as instances of `EngineeringBlogFeed` and `BlogArticle` entities, ensuring updates are idempotent.
import requests
from openmetadata.api import OpenMetadataAPI
api = OpenMetadataAPI('https://your-metadata-instance.com')
# create or update custom entity types
api.create_entity_type(EngineeringBlogFeed)
api.create_entity_type(BlogArticle)
# ingest RSS data into the catalog
for entry in feed.entries:
article = BlogArticle(
title=entry.title,
link=entry.link,
# other properties...
)
api.create_entity(article)
Complete Script
The full runnable script combining all steps:
#!/usr/bin/env python3
import feedparser
from openmetadata.entity import Entity
from openmetadata.api import OpenMetadataAPI
class EngineeringBlogFeed(Entity):
# define properties for the feed entity
pass
class BlogArticle(Entity):
# define properties for the article entity
pass
def ingest_rss_data(feed_url, api_url):
feed = feedparser.parse(feed_url)
api = OpenMetadataAPI(api_url)
# create or update custom entity types
api.create_entity_type(EngineeringBlogFeed)
api.create_entity_type(BlogArticle)
# ingest RSS data into the catalog
for entry in feed.entries:
article = BlogArticle(
title=entry.title,
link=entry.link,
# other properties...
)
api.create_entity(article)
if __name__ == "__main__":
feed_url = 'https://slack.engineering/feed/'
api_url = 'https://your-metadata-instance.com'
ingest_rss_data(feed_url, api_url)
Expected Output
When you run the script, you should see the custom entities and relationships created in your OpenMetadata instance, and the RSS data ingested into the catalog.
Limitations and Tradeoffs
This approach assumes that the RSS feed provides sufficient metadata for the custom entities. If the feed is incomplete or inconsistent, the catalog may not accurately reflect the data. Additionally, this solution is tailored to the Slack Engineering RSS feed and may require modifications to work with other feeds.
Frequently Asked Questions
What is the difference between a custom entity type and a standard entity type in OpenMetadata?
A custom entity type is a user-defined entity type that can be created to represent specific data assets or concepts, whereas a standard entity type is a pre-defined entity type provided by OpenMetadata.
How do I handle errors when ingesting RSS data into the catalog?
You can use try-except blocks to catch and handle exceptions that may occur during the ingestion process, such as network errors or invalid data.
Can I use this approach to catalog other types of external data sources?
Yes, this approach can be adapted to catalog other types of external data sources, such as JSON files or APIs, by defining custom entity types and relationships and using the OpenMetadata API to ingest the data.
What I'd Change
In a production environment, I would consider implementing additional error handling and logging mechanisms to ensure the reliability and scalability of the data ingestion process. Additionally, I would explore ways to automate the creation and updating of custom entity types and relationships to reduce manual effort and improve maintainability.
