Have you ever found yourself staring at a RAG evaluation report, all green lights on faithfulness and answer relevance, yet a nagging feeling persists that something crucial is missing? I certainly have. When building RAG pipelines that rely on rapidly changing external data, like live API feeds, standard metrics often paint an incomplete picture. They might tell you if your LLM hallucinated, but they won't tell you if the "77,232 stars" it cited for the CPython repo was accurate an hour ago, let alone right now. This gap in evaluation can lead to subtly failing systems in production, where the data itself is a moving target. This post is for engineers and data scientists who are past the RAG basics and need to build truly trustworthy, adaptive RAG systems. We'll dive into how to extend Ragas, the excellent RAG evaluation framework, with custom metrics tailored to the unique demands of dynamic API data, using the GitHub Repo API as our real-world example. By the end, you'll understand how to measure not just RAG quality, but also data freshness and numerical accuracy against live sources, and integrate these bespoke checks into your continuous evaluation pipeline.
Key Takeaways
- Ragas can be extended with custom evaluation functions to measure domain-specific performance beyond its built-in metrics.
- Architecting custom metrics for dynamic API data requires comparing RAG output against live API calls to assess freshness and numerical accuracy.
- Integrating custom metrics into the Ragas
evaluatefunction allows for a unified evaluation report and actionable quality gates. - Understanding the tradeoffs between LLM-based and rule-based custom metrics is crucial for cost-effective, production-grade evaluation.
- Continuous evaluation with custom metrics provides the feedback loop necessary to maintain RAG performance in environments with rapidly changing knowledge bases.
The Challenge of Dynamic Data in RAG Evaluation
Our previous discussions on operationalizing RAG focused on establishing continuous evaluation and quality gates. That's a solid foundation. However, when your RAG system's knowledge base isn't static but constantly updated via external APIs, the notion of "ground truth" becomes fluid. How do you evaluate if an answer about a GitHub repository's star count is correct if that count changes every minute? Standard metrics like faithfulness or context precision, while critical, don't inherently account for the temporal accuracy or numerical correctness against a live data source. We need a way to build evaluation hooks that reach out to the very APIs our RAG pipeline consumes, verifying the freshness and precision of generated answers in real-time.
Data and Sources
For this exploration, we'll be interacting with the public GitHub REST API to fetch repository details. Specifically, we'll focus on the CPython repository.
- GitHub API Documentation:
https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#get-a-repository - Ragas Documentation:
https://docs.ragas.io/en/latest/ - LangChain Documentation (for RAG pipeline setup):
https://python.langchain.com/docs/get_started/introduction - OpenAI API documentation (for LLM calls, if used by Ragas or custom metrics):
https://platform.openai.com/docs/api-reference - Previous post: "Operationalizing RAG: Continuous Evaluation and Quality Gates for Dynamic Content Streams":
http://blogs.mausamadhikari.com.np/2026/09/operationalizing-rag-continuous.html
Data accessed on 2024-07-30.
Step 1: Architecting a Minimal RAG Pipeline for GitHub Repo Data
Before we can evaluate, we need something to evaluate. This step sets up a barebones RAG system that simulates fetching GitHub repository data and generating an answer. Our goal here isn't a production-ready RAG, but a functional skeleton to demonstrate the evaluation process. We'll fetch live data from the GitHub API and wrap it in a LangChain Document to simulate our knowledge base.
First, we need a way to fetch the live GitHub data. This will be crucial not only for our RAG context but also for our custom evaluation metric's ground truth.
import requests
import os
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI
def fetch_github_repo_data(owner: str, repo: str) -> dict:
"""Fetches live repository data from GitHub API."""
url = f"https://api.github.com/repos/{owner}/{repo}"
try:
response = requests.get(url, timeout=5)
response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)
return response.json()
except requests.exceptions.HTTPError as e:
print(f"HTTP error fetching repo data for {owner}/{repo}: {e}")
return {}
except requests.exceptions.ConnectionError as e:
print(f"Connection error fetching repo data: {e}")
return {}
except requests.exceptions.Timeout:
print(f"Timeout fetching repo data for {owner}/{repo}")
return {}
except requests.exceptions.RequestException as e:
print(f"An error occurred: {e}")
return {}
def create_rag_context_from_github(repo_data: dict) -> list[Document]:
"""Creates a LangChain Document from GitHub repo data."""
if not repo_data:
return []
content = (
f"Repository Name: {repo_data.get('full_name', 'N/A')}\n"
f"Description: {repo_data.get('description', 'No description provided.')}\n"
f"Stars: {repo_data.get('stargazers_count', 0)}\n"
f"Forks: {repo_data.get('forks_count', 0)}\n"
f"Open Issues: {repo_data.get('open_issues_count', 0)}\n"
f"Last Updated: {repo_data.get('updated_at', 'N/A')}"
)
metadata = {
"source": "GitHub API",
"repo_name": repo_data.get('full_name'),
"updated_at": repo_data.get('updated_at')
}
return [Document(page_content=content, metadata=metadata)]
# Placeholder RAG chain
def get_rag_chain(llm_model: str = "gpt-3.5-turbo-0125"):
"""Returns a simple RAG chain."""
llm = ChatOpenAI(model=llm_model, temperature=0, api_key=os.getenv("OPENAI_API_KEY"))
prompt = ChatPromptTemplate.from_messages([
("system", "You are an expert assistant providing information about GitHub repositories. Answer the question truthfully based on the provided context."),
("user", "Context: {context}\nQuestion: {question}")
])
return prompt | llm | StrOutputParser()
This snippet defines functions to fetch real GitHub data and transform it into a Document. We've also set up a basic LangChain RAG chain. For our simple demonstration, the "retrieval" step will just pass the single relevant document to the LLM.
Step 2: Baseline Evaluation with Ragas's Core Metrics
With our minimal RAG system in place, let's establish a baseline using Ragas's built-in metrics. This will give us a foundational understanding of how our RAG system performs on aspects like faithfulness and answer relevance. It also highlights where these generic metrics might fall short for dynamic data.
