Scaling ETL: When to Trade Pandas for Polars in Your Production Pipelines
As data volumes grow, even the most robust ETL pipelines can become bottlenecks, with Pandas-heavy transformations consuming excessive memory and…
Latest posts
Mitigating LLM Hallucination in Customer-Facing Chatbots: An End-to-End Approach
By implementing a combination of semantic analysis, fact-checking, and user feedback mechanisms, developers can effectively mitigate LLM hallucin…
Building a Real-Time Anomaly Detection System: Lessons from the Netflix Tech Blog
Manually monitoring blog post performance can be a tedious and time-consuming task, especially when dealing with a large volume of data. The Netf…
A/B Testing Pitfalls We Learned the Hard Way: Avoiding Common Mistakes in Your Next Experiment
What's the most common mistake you've made in your A/B testing experiments? For me, it was ignoring confounding variables, which led to f…
Unpacking Remittance Impact on Nepal's Economy: A Data-Driven Analysis
As I delve into the intricacies of Nepal's economy, a question consistently resonates with me: what is the true extent of remittances' in…
Building Resilient Data Pipelines: An Idempotent Approach to External API Ingestion with Airflow
When you're building production data pipelines, especially those consuming data from external APIs, you inevitably hit snags: network flakine…
Detecting Anomalies in Time Series Data: A Real-World Example with Open Library Search API
Have you ever stared at a sea of time series data, wondering what secrets it holds about user behavior, trends, and anomalies? I have, and it'…
Verifying Truth: A Semantic Approach to Detecting Hallucinations in Customer-Facing LLMs
As I delved into the world of customer-facing chatbots, I realized that even with advanced Retrieval Augmented Generation (RAG) techniques, Large…
Scaling Financial Data Scraping in Nepal: A Real-World Example with Discord Engineering Blog Data
As I delved into the world of financial data scraping in Nepal, I was struck by the scarcity of reliable and up-to-date sources. The Nepali finan…
Scraping and Normalizing Nepali Financial Data: A Step-by-Step Guide
What if you could transform the disparate and often unstructured financial data from Nepali sources into a coherent, normalized dataset, empoweri…