Many organizations struggle to provide relevant search experiences for frequently updated content like blogs, documentation, or news feeds. Traditional keyword search often misses conceptual matches, while naive semantic search implementations require costly full re-indexing for every update. This post shows how to build a robust semantic search layer that stays fresh, performs efficiently, and delivers precise results for dynamic data, ideal for developers and data scientists building production-grade information retrieval systems.
Key Takeaways
- Implement incremental indexing with a vector database to efficiently update search indexes for dynamic content without full re-indexing.
- Leverage metadata filtering in conjunction with semantic search to refine results and improve user experience.
- Understand the tradeoffs between local and API-based embedding models for performance, cost, and domain specificity.
The Problem
The problem of maintaining a fresh and relevant search index for dynamic content is a common challenge. Traditional approaches often require significant computational resources and may not provide the best results.
Data and Sources
This post uses the Stripe Blog RSS feed as a real-world example of dynamic content. The feed is accessed through the URL https://stripe.com/blog/feed.rss. The data is fetched using the `feedparser` library, and the embeddings are generated using the `sentence-transformers` library. The vector database used is ChromaDB. Data accessed on 2024-09-16.
Loading the Data
The first step is to fetch the RSS feed and parse it into a format suitable for embedding. This can be achieved using the `feedparser` library.
import feedparser
feed = feedparser.parse('https://stripe.com/blog/feed.rss')
entries = feed.entries
Embedding for Semantic Understanding
The next step is to convert the human-readable text into numerical vector representations that capture semantic meaning. This can be achieved using the `sentence-transformers` library.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = []
for entry in entries:
text = entry.title + ' ' + entry.summary
embedding = model.encode(text)
embeddings.append(embedding)
Persistent Vectors: Setting Up ChromaDB
The embeddings need to be stored in an efficient, queryable structure for fast similarity search. This can be achieved using ChromaDB.
from chromadb import PersistentClient
client = PersistentClient("chromadb")
collection = client.get_collection("stripe_blog")
Implementing Incremental Indexing
To keep the search index fresh, new or modified content needs to be updated in the vector database without requiring a full rebuild.
def update_index(collection, entries):
for entry in entries:
link = entry.link
if collection.get(link):
# Update existing document
text = entry.title + ' ' + entry.summary
embedding = model.encode(text)
collection.upsert(link, embedding)
else:
# Add new document
text = entry.title + ' ' + entry.summary
embedding = model.encode(text)
collection.add(link, embedding)
Precision Search: Leveraging Metadata Filtering
To improve search accuracy and user experience, metadata filtering can be applied in conjunction with semantic search.
def search(collection, query, metadata_filter):
results = collection.query(query, where=metadata_filter)
return results
Complete Script
The full runnable script combining all steps:
#!/usr/bin/env python3
import feedparser
from sentence_transformers import SentenceTransformer
from chromadb import PersistentClient
def load_data():
feed = feedparser.parse('https://stripe.com/blog/feed.rss')
return feed.entries
def embed_data(entries):
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = []
for entry in entries:
text = entry.title + ' ' + entry.summary
embedding = model.encode(text)
embeddings.append(embedding)
return embeddings
def update_index(collection, entries):
model = SentenceTransformer('all-MiniLM-L6-v2')
for entry in entries:
link = entry.link
if collection.get(link):
text = entry.title + ' ' + entry.summary
embedding = model.encode(text)
collection.upsert(link, embedding)
else:
text = entry.title + ' ' + entry.summary
embedding = model.encode(text)
collection.add(link, embedding)
def search(collection, query, metadata_filter):
results = collection.query(query, where=metadata_filter)
return results
if __name__ == "__main__":
client = PersistentClient("chromadb")
collection = client.get_collection("stripe_blog")
entries = load_data()
embeddings = embed_data(entries)
update_index(collection, entries)
query = "semantic search"
metadata_filter = {"published_parsed": {"$gt": "2024-01-01"}}
results = search(collection, query, metadata_filter)
print(results)
Expected Output
The expected output will be a list of relevant search results based on the query and metadata filter.
Limitations and Tradeoffs
While this approach provides an efficient and accurate semantic search experience, there are limitations and tradeoffs to consider. The choice of embedding model can impact performance and cost. The use of a local vector database may not scale to very large datasets. The implementation of incremental indexing and metadata filtering can add complexity to the system.
Frequently Asked Questions
How do I handle updates to existing blog posts in the vector database?
You can handle updates to existing blog posts by checking if the document already exists in the vector database and updating the embedding if necessary.
When should I consider switching from an embedded vector database like ChromaDB to a cloud-hosted solution?
You should consider switching to a cloud-hosted solution when your dataset becomes too large to manage locally or when you need to scale your search application to handle a large number of users.
What if my content is in a language other than English?
Most modern embedding models support multiple languages, but you may need to fine-tune the model for your specific language or use a language-specific model.
What I'd Change
In a production environment, I would consider using a more robust and scalable vector database solution, such as Pinecone or Weaviate, to handle large datasets and high query volumes. I would also consider implementing additional features, such as query caching and result ranking, to further improve the search experience.
