How I Turbocharged a Discord RSS ETL Pipeline with `uv` and `pyproject.toml`

Q: What if the feed adds a new field I care about?

Extend the load_feed loop to capture the new key, and add the column to the DataFrame. Because the script writes a CSV, downstream consumers will automatically see the new column.

Q: Is the lock file safe to commit?

Yes. Commit uv.lock alongside pyproject.toml . It is the single source of truth for versions and hashes.

Q: How do I install only the core dependencies on a CI runner?

Run uv sync without any group flags. To add the plotting extra, use uv sync --group plot .

How I Turbocharged a Discord RSS ETL Pipeline with `uv` and `pyproject.toml`

Ever spent an hour watching `pip install -r requirements.txt` spin forever while a nightly ETL job silently failed because a library version drifted on the production server? I hit that wall last quarter when I tried to pull the Discord Engineering RSS feed into a daily analytics table. The fix wasn’t a new library—it was a switch to `uv` and a disciplined `pyproject.toml`. If you’re a data engineer who’s tired of “works on my laptop” and wants a reproducible, lightning‑fast setup for real‑world feeds, read on. I’ll show you how I built the pipeline, why each piece matters, and the exact commands that turned a flaky script into a rock‑solid production asset.

Key Takeaways

`uv` resolves and installs dependencies up to 20× faster than `pip`, and its lock file guarantees identical environments across dev, CI, and production.
Keeping all metadata in `pyproject.toml` eliminates the split‑brain problem of `requirements.txt` + `setup.py`.
Optional dependency groups let you add heavy extras (e.g., `matplotlib`) only where needed, keeping the core pipeline lightweight.
Explicit error handling for network and parsing failures makes the script safe to run unattended.
Running the script with `uv run` removes the need to pre‑create a virtual environment, simplifying Docker or serverless deployments.

The Problem

My team needed a reliable daily pull of the Discord Engineering blog posts, transform them into a tidy CSV, and generate a quick trend chart. The original prototype used a `requirements.txt` file, a hand‑crafted virtualenv, and a naïve `requests` call. In production the job stalled on dependency resolution, and a silent upgrade of `feedparser` broke the date parsing logic. The goal was to eliminate those hidden costs and make the pipeline reproducible with a single source of truth.

Data and Sources

The feed lives at https://discord.com/blog/rss.xml. I used the public feedparser library to read it, pandas for tabular work, and matplotlib for the optional chart. Official docs:

Data accessed on 2026-10-30.

Loading the Data

The first step is to fetch the RSS XML and turn each entry into a dictionary. I wrap the network call in a try/except block to catch timeouts or HTTP errors, and I validate that the feed contains entries before proceeding.

import feedparser
import requests
from datetime import datetime

def load_feed(url: str) -> list[dict]:
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
    except requests.RequestException as exc:
        raise RuntimeError(f"Failed to download RSS feed: {exc}") from exc

    feed = feedparser.parse(response.content)
    if feed.bozo:
        raise ValueError("Malformed RSS feed")
    if not feed.entries:
        raise RuntimeError("RSS feed returned no entries")

    records = []
    for entry in feed.entries:
        records.append({
            "title": entry.get("title", "").strip(),
            "link": entry.get("link", ""),
            "published": _parse_date(entry.get("published", "")),
        })
    return records

def _parse_date(date_str: str) -> datetime:
    try:
        return datetime.strptime(date_str, "%a, %d %b %Y %H:%M:%S %z")
    except Exception:
        # fallback to UTC now to avoid breaking downstream logic
        return datetime.utcnow()

The Core Logic

With a clean list of dicts, I hand it to pandas, deduplicate by URL, and write a CSV. The optional plotting step lives behind an import guard so the core pipeline stays lightweight when the `plot` extra isn’t installed.

import pandas as pd
from pathlib import Path

def transform_and_save(records: list[dict], out_path: Path) -> pd.DataFrame:
    df = pd.DataFrame(records)
    df.drop_duplicates(subset="link", inplace=True)
    df.sort_values("published", inplace=True)
    out_path.parent.mkdir(parents=True, exist_ok=True)
    df.to_csv(out_path, index=False)
    return df

def optional_plot(df: pd.DataFrame, img_path: Path) -> None:
    try:
        import matplotlib.pyplot as plt
    except ImportError as exc:
        raise RuntimeError("Plotting requires the 'plot' optional dependency") from exc

    # Count posts per month
    monthly = df.set_index("published").resample("M").size()
    plt.figure(figsize=(8, 4))
    monthly.plot(kind="bar", color="#7289da")
    plt.title("Discord Engineering Posts per Month")
    plt.ylabel("Posts")
    plt.tight_layout()
    img_path.parent.mkdir(parents=True, exist_ok=True)
    plt.savefig(img_path)

Putting It Together

The entry point glues loading, transformation, and optional visualization. I catch the two custom error types raised earlier and surface a clean message for CI logs.

import sys
from pathlib import Path

def main() -> None:
    FEED_URL = "https://discord.com/blog/rss.xml"
    CSV_PATH = Path("data/discord_posts.csv")
    IMG_PATH = Path("output/posts_per_month.png")

    try:
        records = load_feed(FEED_URL)
        df = transform_and_save(records, CSV_PATH)
        print(f"✅ Saved {len(df)} rows to {CSV_PATH}")

        # Plot only if the optional group is installed
        if "plot" in sys.argv:
            optional_plot(df, IMG_PATH)
            print(f"🖼️ Chart written to {IMG_PATH}")
    except (RuntimeError, ValueError) as err:
        print(f"❌ Pipeline aborted: {err}", file=sys.stderr)
        sys.exit(1)

if __name__ == "__main__":
    main()

Complete Script

The full runnable script combining all steps. Save it as discord_rss_etl.py and run with uv run discord_rss_etl.py (or python discord_rss_etl.py if you prefer).

#!/usr/bin/env python3
"""
Turbocharged Discord RSS ETL using uv + pyproject.toml

- Fast, deterministic dependency resolution via uv
- Single source of truth in pyproject.toml
- Optional plotting group to keep the core lightweight
"""

import sys
from pathlib import Path
from datetime import datetime

import feedparser
import pandas as pd
import requests

def _parse_date(date_str: str) -> datetime:
    """Parse RFC 2822 dates; fall back to UTC now on failure."""
    try:
        return datetime.strptime(date_str, "%a, %d %b %Y %H:%M:%S %z")
    except Exception:
        return datetime.utcnow()

def load_feed(url: str) -> list[dict]:
    """Download and parse an RSS feed, returning a list of records."""
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
    except requests.RequestException as exc:
        raise RuntimeError(f"Failed to download RSS feed: {exc}") from exc

    feed = feedparser.parse(response.content)
    if feed.bozo:
        raise ValueError("Malformed RSS feed")
    if not feed.entries:
        raise RuntimeError("RSS feed returned no entries")

    records = []
    for entry in feed.entries:
        records.append({
            "title": entry.get("title", "").strip(),
            "link": entry.get("link", ""),
            "published": _parse_date(entry.get("published", "")),
        })
    return records

def transform_and_save(records: list[dict], out_path: Path) -> pd.DataFrame:
    """Deduplicate, sort, and write records to CSV."""
    df = pd.DataFrame(records)
    df.drop_duplicates(subset="link", inplace=True)
    df.sort_values("published", inplace=True)
    out_path.parent.mkdir(parents=True, exist_ok=True)
    df.to_csv(out_path, index=False)
    return df

def optional_plot(df: pd.DataFrame, img_path: Path) -> None:
    """Create a bar chart of posts per month; requires the 'plot' extra."""
    try:
        import matplotlib.pyplot as plt
    except ImportError as exc:
        raise RuntimeError("Plotting requires the 'plot' optional dependency") from exc

    monthly = df.set_index("published").resample("M").size()
    plt.figure(figsize=(8, 4))
    monthly.plot(kind="bar", color="#7289da")
    plt.title("Discord Engineering Posts per Month")
    plt.ylabel("Posts")
    plt.tight_layout()
    img_path.parent.mkdir(parents=True, exist_ok=True)
    plt.savefig(img_path)

def main() -> None:
    FEED_URL = "https://discord.com/blog/rss.xml"
    CSV_PATH = Path("data/discord_posts.csv")
    IMG_PATH = Path("output/posts_per_month.png")

    try:
        records = load_feed(FEED_URL)
        df = transform_and_save(records, CSV_PATH)
        print(f"✅ Saved {len(df)} rows to {CSV_PATH}")

        # Plot only when the optional group is requested
        if "plot" in sys.argv:
            optional_plot(df, IMG_PATH)
            print(f"🖼️ Chart written to {IMG_PATH}")
    except (RuntimeError, ValueError) as err:
        print(f"❌ Pipeline aborted: {err}", file=sys.stderr)
        sys.exit(1)

if __name__ == "__main__":
    main()

Expected Output

When you run uv run discord_rss_etl.py you should see something like:

✅ Saved 27 rows to data/discord_posts.csv
🖼️ Chart written to output/posts_per_month.png

The CSV will contain columns title, link, and published. The PNG visualizes the monthly posting frequency.

Limitations and Tradeoffs

This approach assumes the RSS feed is publicly reachable and fits in memory. For multi‑GB streams you’d need a streaming parser or chunked writes. The optional plotting group adds a runtime dependency; if you ship the script to a minimal container you must remember to include the extra (e.g., uv sync --group plot). Finally, `uv` is still young—some obscure wheels may lack pre‑built binaries, requiring a fallback to `pip` in those edge cases.

Frequently Asked Questions

Why not stick with pip and a requirements.txt?

`pip` resolves dependencies sequentially and can produce nondeterministic builds. `uv` generates a lock file that records exact hashes, guaranteeing the same environment everywhere.

Can I use this pipeline inside Docker?

Absolutely. The Dockerfile only needs to install `uv` (a single binary) and then run uv sync to populate the image layer with a reproducible environment.

What if the feed adds a new field I care about?

Extend the load_feed loop to capture the new key, and add the column to the DataFrame. Because the script writes a CSV, downstream consumers will automatically see the new column.

Is the lock file safe to commit?

Yes. Commit uv.lock alongside pyproject.toml. It is the single source of truth for versions and hashes.

How do I install only the core dependencies on a CI runner?

Run uv sync without any group flags. To add the plotting extra, use uv sync --group plot.

What I'd Change

If I were to iterate again, I’d split the fetch‑transform‑load steps into separate commands orchestrated by a lightweight task runner (e.g., hatch or a Makefile). That would let me cache the raw XML, re‑run only the transformation when the schema changes, and keep CI builds sub‑minute. In short, `uv` solves the environment pain, but a modular command layer is the next frontier for truly maintainable production pipelines.

Py Data