Building a Unified Data Catalog with OpenMetadata and Apache Atlas As a data engineer working with multiple data sources, I've often struggled with data discovery, governance, and management due to the comple…
Uncovering Insights from Nepal Rastra Bank Quarterly Reports: A Step-by-Step Web Scraping Guide Financial analysts and data scientists in Nepal often struggle to access and analyze the quarterly reports published by the Nepal Rastra Bank, wh…
Choosing the Right File Format for Your Data Lake: A Deep Dive into Parquet, Avro, and ORC As a data engineer, have you ever struggled to choose the right file format for your data lake, only to find that your choice led to inefficient …
Mastering Data Cataloging: How to Streamline Data Discovery with OpenMetadata and Apache Atlas Have you ever found yourself drowning in a sea of data, struggling to find the right dataset or understand the context of a particular piece of i…
Building Data Pipelines: A Practical Guide to ETL vs. ELT Architectures for API Integrations Data engineers and data scientists frequently encounter the architectural dilemma of ETL versus ELT when integrating data from external APIs. Thi…
Batch Processing F1 Racing Data with Apache Spark and PySpark: A Production Example As a data engineer working with large F1 racing datasets, you're likely no stranger to the challenges of processing and analyzing this data e…
Ensuring Data Quality at Scale: Advanced Patterns with Great Expectations As a data engineer, I've often struggled to ensure data quality, particularly when working with large datasets or complex data pipelines. Rec…
Scraping and Normalizing Nepali Financial Data: Lessons from Nepal Stock Exchange (NEPSE) As a data analyst or investor interested in the Nepali financial market, you may have encountered the challenge of accessing and analyzing financ…
Catching Data Drift Before It Bites: A Great Expectations Playbook for RSS Feeds The most insidious data problems aren't the ones that break your pipeline; they're the subtle shifts in data quality that go unnoticed, s…
Scaling ETL: When to Trade Pandas for Polars in Your Production Pipelines As data volumes grow, even the most robust ETL pipelines can become bottlenecks, with Pandas-heavy transformations consuming excessive memory and…
Building Resilient Data Pipelines: An Idempotent Approach to External API Ingestion with Airflow When you're building production data pipelines, especially those consuming data from external APIs, you inevitably hit snags: network flakine…