Have you ever spent hours manually sifting through Nepal Rastra Bank's (NRB) Quarterly Economic Bulletins, painstakingly copying numbers from PDF tables into a spreadsheet, only to realize a new report just dropped with a slightly different layout? I certainly have. That feeling of critical macroeconomic insights being trapped behind an inconsistent, unyielding PDF barrier is a common frustration for financial analysts and data scientists in Nepal. It slows down market reactions, introduces human error, and makes automated analysis a pipe dream. What if we could build a system that not only extracts this data but does so reliably, even when the reports throw curveballs? In this post, I'll walk you through the resilient Python pipeline I built to tackle this exact problem, showing you how to programmatically discover, download, extract, clean, and standardize tabular financial data from these challenging PDF documents, transforming raw reports into actionable intelligence for your models and dashboards.
Key Takeaways
- Leverage web scraping with `BeautifulSoup` to dynamically discover and resolve PDF report links from government websites, building a robust entry point for your data pipeline.
- Implement local caching strategies to enhance pipeline resilience, reduce network load, and prevent repeated downloads of static PDF resources.
- Master `tabula-py` for precision table extraction from complex, multi-page PDFs, even when dealing with varied layouts and merged cells.
- Develop flexible data standardization techniques using `pandas` to unify heterogeneous table structures and data types across different report versions.
- Design a pipeline with comprehensive error handling to gracefully manage network failures, malformed PDFs, and unexpected data formats, ensuring continuous operation.
The Problem: Unlocking NRB's Macroeconomic Data
For anyone tracking Nepal's economy, the Nepal Rastra Bank's Quarterly Economic Bulletins are goldmines of information. They contain crucial indicators on inflation, money supply, balance of payments, and more. The challenge isn't access to the reports themselves – they're publicly available – but rather getting the data out of their PDF format and into a structured database or DataFrame where it can be analyzed. These aren't simple, machine-readable PDFs; they often involve scanned text, inconsistent table formatting across quarters, and tables spanning multiple pages. My goal was to automate this, building a system that could reliably pull, clean, and standardize this data with minimal manual intervention, turning a tedious chore into a repeatable, scalable process.
Data and Sources
Our primary data source is the official Nepal Rastra Bank (NRB) website, specifically their Quarterly Economic Bulletin archive page. This page lists various publications, including the Quarterly Economic Bulletins, with direct links to PDF files. We'll be scraping this page to find the PDF URLs and then downloading the PDFs for extraction.
Data accessed on 2024-07-28.
Step 1 — Discovering and Resolving Report Links
The first hurdle is finding the PDF links. The NRB website is a standard HTML page, so a simple HTTP request and HTML parsing with `BeautifulSoup` is our weapon of choice. The key is to identify a consistent pattern to locate the Quarterly Economic Bulletin links amidst other publications. I observed that these links usually contain "Quarterly Economic Bulletin" in their text or URL and point to `.pdf` files.
import requests
from bs4 import BeautifulSoup
import os
import re
from urllib.parse import urljoin
def get_nrb_pdf_links(base_url: str) -> dict[str, str]:
"""
Fetches the NRB publications page and extracts links to Quarterly Economic Bulletins.
Returns a dictionary mapping bulletin titles to their full PDF URLs.
"""
try:
response = requests.get(base_url, timeout=10)
response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)
except requests.exceptions.RequestException as e:
print(f"Error fetching {base_url}: {e}")
return {}
soup = BeautifulSoup(response.content, 'html.parser')
pdf_links = {}
# Look for tags that contain "Quarterly Economic Bulletin" and link to a PDF
for a_tag in soup.find_all('a', href=True):
link_text = a_tag.get_text(strip=True)
href = a_tag['href']
# Check if the text or the href indicates a Quarterly Economic Bulletin PDF
if "Quarterly Economic Bulletin" in link_text and href.endswith('.pdf'):
full_url = urljoin(base_url, href)
# Clean up the title a bit, removing excess whitespace or 'pdf'
title = link_text.replace('Quarterly Economic Bulletin -', '').strip()
pdf_links[title] = full_url
elif "Quarterly Economic Bulletin" in href and href.endswith('.pdf'):
full_url = urljoin(base_url, href)
# Try to derive a title from the URL if text is absent or generic
title_match = re.search(r'QEB_(\d{4})_(\w+)\.pdf', href, re.IGNORECASE)
if title_match:
title = f"Quarterly Economic Bulletin {title_match.group(1)} {title_match.group(2)}"
else:
title = os.path.basename(href).replace('.pdf', '').replace('_', ' ').strip()
pdf_links[title] = full_url
return pdf_links
# Example usage (not part of the final script's main execution, just for illustration)
# NRB_PUBLICATIONS_URL = "https://www.nrb.org.np/contents/publications/economic-bulletin/"
# links = get_nrb_pdf_links(NRB_PUBLICATIONS_URL)
# print(f"Found {len(links)} PDF links.")
# for title, url in list(links.items())[:2]:
# print(f"- {title}: {url}")
The `get_nrb_pdf_links` function sends an HTTP GET request to the NRB publications page. It then parses the HTML content with `BeautifulSoup`, looking for anchor (``) tags that meet our criteria. I've added a `try-except` block to catch network issues, ensuring our pipeline doesn't crash on transient connection problems. The `urljoin` function is crucial for constructing absolute URLs from relative paths found in the HTML.
Step 2 — Resilient PDF Download and Local Caching
Downloading PDFs repeatedly for every run is inefficient and puts unnecessary load on the NRB server. A local cache is essential. This step involves checking if a PDF already exists locally before attempting to download it. If it doesn't, we download it; if it does, we simply use the cached version.
import requests
import os
import hashlib # For creating unique filenames
def download_pdf(pdf_url: str, cache_dir: str = "nrb_pdf_cache") -> str | None:
"""
Downloads a PDF from a given URL, caching it locally.
Returns the path to the downloaded/cached PDF, or None on failure.
"""
os.makedirs(cache_dir, exist_ok=True)
# Use a hash of the URL to create a unique, safe filename for caching
filename = hashlib.md5(pdf_url.encode('utf-8')).hexdigest() + ".pdf"
file_path = os.path.join(cache_dir, filename)
if os.path.exists(file_path):
print(f"Using cached PDF: {file_path}")
return file_path
print(f"Downloading {pdf_url} to {file_path}...")
try:
response = requests.get(pdf_url, stream=True, timeout=30)
response.raise_for_status()
with open(file_path, 'wb') as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
print(f"Downloaded: {file_path}")
return file_path
except requests.exceptions.RequestException as e:
print(f"Error downloading {pdf_url}: {e}")
# Clean up potentially incomplete download
