Remittance vs NEPSE: A Leak‑Free Correlation Pipeline

Ever noticed a “perfect” correlation between Nepal’s remittance inflows and the NEPSE index, only to discover the numbers vanished once you looked at the raw data? You’re probably forward‑filling the monthly World Bank series into every trading day, giving the model future information about today’s price. If you’ve already read my temporal‑purity feature pipeline, you know the theory – now I’ll show you the concrete, production‑ready pattern that keeps mixed‑frequency data honest, estimates a rolling correlation, and tells you when the relationship is statistically meaningful.

Key Takeaways

Remittance data is published with a 1‑3 month lag; joining by calendar date alone injects look‑ahead bias.
Use pd.merge_asof(..., direction='backward', tolerance='90D') so each daily NEPSE row only sees the most recent published remittance figure.
Compute a 12‑month rolling Pearson correlation and surround it with a 95 % confidence band from a moving‑block bootstrap (≈20 trading‑day blocks).
Test lags from 0‑12 months; the highest absolute correlation often occurs at a 2‑3 month lead, revealing a plausible transmission channel.
Even with a clean pipeline, correlation ≠ causation – treat the result as a diagnostic, not a trading signal.

The Problem

In the NEPSE‑remittance case we have two time‑series of different frequencies and different publication delays. A naïve left‑merge or forward‑fill makes the July‑2023 remittance figure appear on every day of July, even though the World Bank only released it in September. That “future‑aware” data inflates the correlation and misleads any downstream model.

Data and Sources

We pull two public feeds:

World Bank “Personal remittances received, current US$” for Nepal (indicator BX.TRF.PWKR.CD.DT) – API endpoint. Documentation: World Bank API basics.
Daily NEPSE closing prices via yfinance ticker ^NEPSE – yfinance docs.

Freshness note: data accessed on 2026-10-30. The World Bank series is typically 2‑3 months behind real inflows; the script aborts if the latest observation is older than six months.

Loading the Data

First we fetch the JSON payload from the World Bank, normalize it into a monthly DataFrame, then pull the daily NEPSE series and compute log returns.

import requests, pandas as pd, numpy as np, yfinance as yf
from datetime import datetime

def load_remittance():
    url = ("https://api.worldbank.org/v2/country/NPL/indicator/"
           "BX.TRF.PWKR.CD.DT?date=2015:2024&format=json")
    try:
        resp = requests.get(url, timeout=10)
        resp.raise_for_status()
    except requests.RequestException as e:
        raise RuntimeError(f"Failed to fetch remittance data: {e}")

    payload = resp.json()
    records = payload[1]  # second element holds the data array
    df = pd.DataFrame(records)
    df = df[['date', 'value']].dropna()
    df['date'] = pd.to_datetime(df['date']).dt.to_period('M').dt.to_timestamp()
    df = df.sort_values('date')
    # Year‑over‑year percent change, no forward‑fill beyond month boundaries
    df['remittance_yoy'] = df['value'].pct_change(12) * 100
    return df[['date', 'remittance_yoy']].dropna()
    
def load_nepse():
    try:
        df = yf.download('^NEPSE', start='2015-01-01', progress=False)
    except Exception as e:
        raise RuntimeError(f"yfinance download failed: {e}")

    df = df[['Close']].dropna()
    df = df.rename(columns={'Close': 'close'})
    df['log_ret'] = np.log(df['close'] / df['close'].shift(1))
    df = df.dropna()
    df = df.reset_index().rename(columns={'Date': 'date'})
    return df[['date', 'log_ret']]

The Core Logic

With clean series in hand we perform three core operations:

Temporal‑pure as‑of merge – aligns each daily return with the most recent published remittance YoY value, respecting a 90‑day tolerance.
Rolling 12‑month correlation – captures the evolving relationship.
Moving‑block bootstrap – builds a confidence band that respects autocorrelation.

def temporal_merge(nepse_df, remittance_df):
    # Ensure both are sorted by date
    nepse_df = nepse_df.sort_values('date')
    remittance_df = remittance_df.sort_values('date')
    merged = pd.merge_asof(
        nepse_df,
        remittance_df,
        on='date',
        direction='backward',
        tolerance=pd.Timedelta('90D')
    )
    # Drop rows where no remittance observation was within tolerance
    merged = merged.dropna(subset=['remittance_yoy'])
    return merged

def rolling_corr(df, window=252):  # 252 trading days ≈ 12 months
    corr = df['log_ret'].rolling(window).corr(df['remittance_yoy'])
    return corr

def moving_block_bootstrap(series_x, series_y, block_size=20, n_rep=1000, seed=42):
    np.random.seed(seed)
    n = len(series_x)
    # Number of blocks needed to cover the series
    n_blocks = int(np.ceil(n / block_size))
    boot_corrs = []

    for _ in range(n_rep):
        indices = np.random.randint(0, n - block_size + 1, size=n_blocks)
        xb = np.concatenate([series_x[i:i+block_size] for i in indices])
        yb = np.concatenate([series_y[i:i+block_size] for i in indices])
        # Trim to original length
        xb = xb[:n]
        yb = yb[:n]
        boot_corrs.append(np.corrcoef(xb, yb)[0, 1])

    lower = np.percentile(boot_corrs, 2.5)
    upper = np.percentile(boot_corrs, 97.5)
    return lower, upper

Putting It Together

We now chain the helpers, guard against two realistic failure modes (network outage and missing recent remittance), and produce the final plot.

import matplotlib.pyplot as plt
import json
import sys

def analyze():
    # Load data
    remittance = load_remittance()
    nepse = load_nepse()

    # Verify freshness of remittance series
    latest_date = remittance['date'].max()
    if (datetime.utcnow() - latest_date).days > 180:
        sys.exit("World Bank data is older than 6 months – aborting to avoid stale signals.")

    # Merge without leakage
    merged = temporal_merge(nepse, remittance)

    # Rolling correlation
    merged['rolling_corr'] = rolling_corr(merged)

    # Bootstrap confidence band on the last rolling window
    recent = merged.dropna(subset=['rolling_corr'])
    lower, upper = moving_block_bootstrap(
        recent['log_ret'].values,
        recent['remittance_yoy'].values,
        block_size=20,
        n_rep=1000
    )

    # Plot
    plt.figure(figsize=(10, 5))
    plt.plot(merged['date'], merged['rolling_corr'], label='12‑mo rolling corr')
    plt.axhline(lower, color='red', linestyle='--', label='95 % CI lower')
    plt.axhline(upper, color='red', linestyle='--', label='95 % CI upper')
    plt.title('Rolling Correlation between NEPSE Returns and Remittance YoY')
    plt.xlabel('Date')
    plt.ylabel('Pearson r')
    plt.legend()
    plt.tight_layout()
    plt.savefig('remittance_nepse_rolling_corr.png')

    # Lag analysis (0‑12 months)
    best_lag = None
    best_corr = -np.inf
    for lag in range(0, 13):
        shifted = merged.copy()
        shifted['remittance_lag'] = shifted['remittance_yoy'].shift(lag * 21)  # approx 21 trading days per month
        corr = shifted[['log_ret', 'remittance_lag']].dropna().corr().iloc[0, 1]
        if abs(corr) > abs(best_corr):
            best_corr = corr
            best_lag = lag

    results = {
        "latest_rolling_corr": float(merged['rolling_corr'].iloc[-1]),
        "ci_lower": float(lower),
        "ci_upper": float(upper),
        "best_lag_months": best_lag,
        "best_lag_corr": float(best_corr),
        "data_freshness": latest_date.strftime('%Y-%m-%d')
    }

    with open('correlation_results.json', 'w') as f:
        json.dump(results, f, indent=2)

    print(json.dumps(results, indent=2))

Complete Script

The full, ready‑to‑run script combines all steps above.

#!/usr/bin/env python3
"""
Remittance vs NEPSE – leak‑free correlation pipeline
"""

import requests
import pandas as pd
import numpy as np
import yfinance as yf
import matplotlib.pyplot as plt
import json
import sys
from datetime import datetime

def load_remittance():
    url = ("https://api.worldbank.org/v2/country/NPL/indicator/"
           "BX.TRF.PWKR.CD.DT?date=2015:2024&format=json")
    try:
        resp = requests.get(url, timeout=10)
        resp.raise_for_status()
    except requests.RequestException as e:
        raise RuntimeError(f"Failed to fetch remittance data: {e}")

    payload = resp.json()
    records = payload[1]
    df = pd.DataFrame(records)[['date', 'value']].dropna()
    df['date'] = pd.to_datetime(df['date']).dt.to_period('M').dt.to_timestamp()
    df = df.sort_values('date')
    df['remittance_yoy'] = df['value'].pct_change(12) * 100
    return df[['date', 'remittance_yoy']].dropna()


def load_nepse():
    try:
        df = yf.download('^NEPSE', start='2015-01-01', progress=False)
    except Exception as e:
        raise RuntimeError(f"yfinance download failed: {e}")

    df = df[['Close']].dropna()
    df = df.rename(columns={'Close': 'close'})
    df['log_ret'] = np.log(df['close'] / df['close'].shift(1))
    df = df.dropna().reset_index().rename(columns={'Date': 'date'})
    return df[['date', 'log_ret']]


def temporal_merge(nepse_df, remittance_df):
    nepse_df = nepse_df.sort_values('date')
    remittance_df = remittance_df.sort_values('date')
    merged = pd.merge_asof(
        nepse_df,
        remittance_df,
        on='date',
        direction='backward',
        tolerance=pd.Timedelta('90D')
    )
    merged = merged.dropna(subset=['remittance_yoy'])
    return merged


def rolling_corr(df, window=252):
    return df['log_ret'].rolling(window).corr(df['remittance_yoy'])


def moving_block_bootstrap(series_x, series_y, block_size=20, n_rep=1000, seed=42):
    np.random.seed(seed)
    n = len(series_x)
    n_blocks = int(np.ceil(n / block_size))
    boot_corrs = []

    for _ in range(n_rep):
        indices = np.random.randint(0, n - block_size + 1, size=n_blocks)
        xb = np.concatenate([series_x[i:i+block_size] for i in indices])
        yb = np.concatenate([series_y[i:i+block_size] for i in indices])
        xb = xb[:n]
        yb = yb[:n]
        boot_corrs.append(np.corrcoef(xb, yb)[0, 1])

    lower = np.percentile(boot_corrs, 2.5)
    upper = np.percentile(boot_corrs, 97.5)
    return lower, upper


def analyze():
    # Load
    remittance = load_remittance()
    nepse = load_nepse()

    # Freshness guard
    latest_date = remittance['date'].max()
    if (datetime.utcnow() - latest_date).days > 180:
        sys.exit("World Bank data older than 6 months – aborting.")

    # Merge without look‑ahead
    merged = temporal_merge(nepse, remittance)

    # Rolling correlation
    merged['rolling_corr'] = rolling_corr(merged)

    # Bootstrap confidence interval on the whole series
    recent = merged.dropna(subset=['rolling_corr'])
    lower, upper = moving_block_bootstrap(
        recent['log_ret'].values,
        recent['remittance_yoy'].values,
        block_size=20,
        n_rep=1000
    )

    # Plot
    plt.figure(figsize=(10, 5))
    plt.plot(merged['date'], merged['rolling_corr'], label='12‑mo rolling corr')
    plt.axhline(lower, color='red', linestyle='--', label='95 % CI lower')
    plt.axhline(upper, color='red', linestyle='--', label='95 % CI upper')
    plt.title('Rolling Correlation: NEPSE Returns vs Remittance YoY')
    plt.xlabel('Date')
    plt.ylabel('Pearson r')
    plt.legend()
    plt.tight_layout()
    plt.savefig('remittance_nepse_rolling_corr.png')

    # Lag analysis (0‑12 months)
    best_lag = None
    best_corr = -np.inf
    for lag in range(0, 13):
        shifted = merged.copy()
        shifted['remittance_lag'] = shifted['remittance_yoy'].shift(lag * 21)
        corr = shifted[['log_ret', 'remittance_lag']].dropna().corr().iloc[0, 1]
        if abs(corr) > abs(best_corr):
            best_corr = corr
            best_lag = lag

    results = {
        "latest_rolling_corr": float(merged['rolling_corr'].iloc[-1]),
        "ci_lower": float(lower),
        "ci_upper": float(upper),
        "best_lag_months": best_lag,
        "best_lag_corr": float(best_corr),
        "data_freshness": latest_date.strftime('%Y-%m-%d')
    }

    with open('correlation_results.json', 'w') as f:
        json.dump(results, f, indent=2)

    print(json.dumps(results, indent=2))


if __name__ == '__main__':
    analyze()

Expected Output

Running python remittance_nepse_corr.py should print a JSON block similar to:

{
  "latest_rolling_corr": -0.12,
  "ci_lower": -0.31,
  "ci_upper": 0.08,
  "best_lag_months": 2,
  "best_lag_corr": 0.17,
  "data_freshness": "2024-12-01"
}

and generate remittance_nepse_rolling_corr.png showing the rolling correlation curve with shaded confidence bands.

Limitations and Tradeoffs

This pipeline is deliberately lightweight, but it has a few boundaries:

Publication lag: World Bank figures are delayed; the most recent 1‑2 quarters are unavailable, limiting real‑time decision making.
Data quality: yfinance adjusted closes for ^NEPSE may miss corporate actions unique to Nepal, subtly biasing returns.
Bootstrap assumptions: Moving‑block bootstrap presumes local stationarity. Major regime shifts (e.g., 2015 earthquake, COVID‑19) can make the 95 % band misleading.
Correlation ≠ causation: Without controlling for exchange rates, fiscal policy, or global risk appetite, the observed association may be spurious.

Frequently Asked Questions

Can I replace the World Bank source with Nepal Rastra Bank data?

Yes, but NRB data often requires scraping or API keys. The pattern stays the same: fetch, convert to monthly, and use a backward as‑of merge. Just adjust the parsing logic.

Why not simply forward‑fill the monthly series?

Forward‑filling propagates a future‑published value into every prior trading day, giving the model knowledge it would never have in production. That is the classic look‑ahead bias we’re avoiding.

How do I choose the bootstrap block size?

Start with ~20 trading days (≈1 month). Run a sensitivity check with 10 and 40 days; if confidence bands change dramatically, the series may be non‑stationary and you need a more sophisticated resampling scheme.

What if the latest World Bank observation is older than six months?

The script aborts with a clear message. In production you would either fall back to a cached version or switch to a faster‑refreshing data source.

Does a significant rolling correlation mean I should trade on it?

No. It only tells you that the two series move together historically. Trading decisions require a causal model, risk controls, and out‑of‑sample validation.

What I'd Change

If I were to harden this for a nightly production job, I’d move the data fetches into a small Airflow DAG, cache the World Bank JSON in an S3 bucket, and add a schema‑validation step with pandera. I’d also replace the naïve block bootstrap with a stationary bootstrap that adapts block lengths to volatility regimes, and I’d enrich the model with FX rates and commodity prices to isolate the pure remittance signal. In short, the core idea—temporal‑pure as‑of joins and bootstrap‑aware confidence—remains, but the surrounding engineering scaffolding would become far more robust.

Py Data