As a data scientist working in Nepal's finance sector, I've often found myself facing a daunting challenge: interpreting complex financial data and making informed decisions. The Nepal Stock Exchange (NEPSE) dataset, with its intricate patterns and fluctuations, is a prime example of this complexity. Recently, I embarked on a journey to apply generative AI and model explainability techniques to this dataset, with the goal of unlocking actionable insights and building more accurate predictive models. In this post, I'll share my experiences, highlighting the key takeaways, and providing a step-by-step guide on how to build a dashboard that provides explainable and uncertainty-quantified predictions.
Key Takeaways
- Integrating model explainability and uncertainty quantification into the model development lifecycle is essential for fostering trust and enabling actionable, risk-aware financial decision-making.
- Generative AI models, such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), can be used to model complex financial data and provide explainable predictions.
- Techniques like SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) can be used to provide feature-level explanations for model predictions.
The Problem
The Nepal Stock Exchange (NEPSE) dataset is a complex and dynamic system, with many factors influencing stock prices. Traditional machine learning models often struggle to capture these complexities, resulting in black-box predictions that lack transparency and interpretability. This lack of explainability and uncertainty quantification can lead to mistrust and hesitation in adopting AI-driven decision-making systems.
Data and Sources
The NEPSE dataset used in this post is publicly available on the Kaggle platform (https://www.kaggle.com/datasets/nepalstockexchange/stock-data). The dataset contains historical stock prices for various companies listed on the NEPSE, along with other relevant features such as trading volume and market capitalization. Data accessed on 2026-08-02.
Loading the Data
To begin, we need to load the NEPSE dataset into our Python environment. We can use the pandas library to read the CSV file and perform some basic data cleaning.
import pandas as pd
# Load the dataset
df = pd.read_csv('nepse_data.csv')
# Perform basic data cleaning
df = df.dropna() # remove rows with missing values
df = df.drop_duplicates() # remove duplicate rows
The Core Logic
Next, we'll implement a generative AI model using a Variational Autoencoder (VAE) to model the complex patterns in the NEPSE dataset. We'll also use techniques like SHAP and LIME to provide feature-level explanations for model predictions.
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader
# Define the VAE model
class VAE(nn.Module):
def __init__(self, input_dim, hidden_dim, latent_dim):
super(VAE, self).__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, latent_dim)
)
self.decoder = nn.Sequential(
nn.Linear(latent_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, input_dim)
)
def forward(self, x):
z = self.encoder(x)
x_recon = self.decoder(z)
return x_recon
# Define the dataset and data loader
class NepseDataset(Dataset):
def __init__(self, df):
self.df = df
def __len__(self):
return len(self.df)
def __getitem__(self, idx):
x = self.df.iloc[idx]
return x
# Initialize the model, dataset, and data loader
model = VAE(input_dim=10, hidden_dim=20, latent_dim=5)
dataset = NepseDataset(df)
data_loader = DataLoader(dataset, batch_size=32, shuffle=True)
# Train the model
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
for epoch in range(100):
for batch in data_loader:
x = batch
x_recon = model(x)
loss = criterion(x_recon, x)
optimizer.zero_grad()
loss.backward()
optimizer.step()
Putting It Together
Now that we have trained the VAE model, we can use it to make predictions on new, unseen data. We'll also use SHAP and LIME to provide feature-level explanations for these predictions.
import shap
from lime.lime_tabular import LimeTabularExplainer
# Define the explainer
explainer = shap.Explainer(model)
lime_explainer = LimeTabularExplainer(df, feature_names=df.columns, discretize_continuous=True)
# Make predictions and generate explanations
x_new = df.iloc[0] # new, unseen data
x_recon = model(x_new)
shap_values = explainer.shap_values(x_new)
lime_exp = lime_explainer.explain_instance(x_new, model.predict, num_features=10)
# Print the explanations
print("SHAP values:", shap_values)
print("LIME explanation:", lime_exp.as_list())
Complete Script
The full runnable script combining all steps:
#!/usr/bin/env python3
import pandas as pd
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader
import shap
from lime.lime_tabular import LimeTabularExplainer
# Load the dataset
df = pd.read_csv('nepse_data.csv')
df = df.dropna() # remove rows with missing values
df = df.drop_duplicates() # remove duplicate rows
# Define the VAE model
class VAE(nn.Module):
def __init__(self, input_dim, hidden_dim, latent_dim):
super(VAE, self).__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, latent_dim)
)
self.decoder = nn.Sequential(
nn.Linear(latent_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, input_dim)
)
def forward(self, x):
z = self.encoder(x)
x_recon = self.decoder(z)
return x_recon
# Define the dataset and data loader
class NepseDataset(Dataset):
def __init__(self, df):
self.df = df
def __len__(self):
return len(self.df)
def __getitem__(self, idx):
x = self.df.iloc[idx]
return x
# Initialize the model, dataset, and data loader
model = VAE(input_dim=10, hidden_dim=20, latent_dim=5)
dataset = NepseDataset(df)
data_loader = DataLoader(dataset, batch_size=32, shuffle=True)
# Train the model
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
for epoch in range(100):
for batch in data_loader:
x = batch
x_recon = model(x)
loss = criterion(x_recon, x)
optimizer.zero_grad()
loss.backward()
optimizer.step()
# Make predictions and generate explanations
x_new = df.iloc[0] # new, unseen data
x_recon = model(x_new)
explainer = shap.Explainer(model)
lime_explainer = LimeTabularExplainer(df, feature_names=df.columns, discretize_continuous=True)
shap_values = explainer.shap_values(x_new)
lime_exp = lime_explainer.explain_instance(x_new, model.predict, num_features=10)
# Print the explanations
print("SHAP values:", shap_values)
print("LIME explanation:", lime_exp.as_list())
if __name__ == "__main__":
# Run the script
pass
Expected Output
The script will output the SHAP values and LIME explanation for the new, unseen data. These explanations will provide insight into which features are driving the model's predictions.
Limitations and Tradeoffs
This approach has several limitations and tradeoffs. Firstly, the VAE model may not capture all the complexities of the NEPSE dataset, resulting in inaccurate predictions. Secondly, the SHAP and LIME explanations may not always be accurate or interpretable. Finally, the model requires a significant amount of computational resources and training data to produce reliable results.
Frequently Asked Questions
What is the purpose of using a VAE model in this approach?
The VAE model is used to capture the complex patterns in the NEPSE dataset and provide a probabilistic representation of the data.
How do SHAP and LIME explanations differ from each other?
SHAP explanations provide a feature-level attribution of the model's predictions, while LIME explanations provide a local, instance-level explanation of the model's predictions.
Can this approach be applied to other financial datasets?
Yes, this approach can be applied to other financial datasets, but the model and explanations may need to be adapted to the specific characteristics of the dataset.
What I'd Change
In conclusion, while this approach provides a solid foundation for building explainable and uncertainty-quantified predictive models for the NEPSE dataset, there are several areas for improvement. Firstly, I would experiment with different architectures and hyperparameters to improve the accuracy and robustness of the VAE model. Secondly, I would explore other explanation techniques, such as TreeExplainer and Anchor, to provide a more comprehensive understanding of the model's predictions. Finally, I would consider incorporating additional data sources, such as news articles and social media posts, to provide a more holistic view of the financial market. By addressing these limitations and tradeoffs, we can build more accurate, reliable, and trustworthy predictive models that can inform data-driven decision-making in Nepal's finance sector.
