As a data scientist working in the Nepali financial sector, I've often encountered the challenge of extracting valuable insights from unstructured data such as news articles and reports. Manual feature engineering is not only time-consuming but also prone to missing subtle contextual nuances. Recently, I explored the potential of combining Large Language Models (LLMs) with AutoML frameworks to automate feature engineering and improve financial predictions. In this post, I'll share my experience and provide a step-by-step guide on how to implement this approach.
Key Takeaways
- Master the integration of LLM-generated features with tabular financial data to enhance predictive models.
- Learn to automatically extract nuanced contextual features from text using `transformers` pipelines.
- Understand how to leverage AutoML frameworks like `AutoGluon` to efficiently build and optimize predictive models.
- Discover strategies for aligning disparate data sources (time series and unstructured text) for robust modeling.
- Gain insights into evaluating and interpreting the impact of LLM-derived features on model performance.
The Problem
Financial analysts and data scientists in markets like Nepal often face challenges in extracting actionable insights from unstructured data and rapidly building performant predictive models. This post aims to address these challenges by demonstrating how to automate feature engineering using LLMs and AutoML frameworks.
Data and Sources
The data used in this example includes historical stock prices from the Nepal Stock Exchange (NEPSE) and financial news headlines. The NEPSE data can be obtained from the NEPSE website, while the financial news headlines can be scraped from online news sources. For this example, we'll use a pre-scraped dataset available at financial_news_headlines.csv. The `AutoGluon` documentation is available at auto.gluon.ai, and the `Hugging Face Transformers` library documentation can be found at huggingface.co. Data accessed on 2023-02-15.
Loading the Data
We'll start by loading the historical stock prices and financial news headlines into a pandas DataFrame.
import pandas as pd
import requests
# Load NEPSE data
nepse_data = pd.read_csv("nepse_data.csv")
# Load financial news headlines
news_headlines = pd.read_csv("financial_news_headlines.csv")
Automating Feature Engineering with LLMs
We'll use the `transformers` library to generate features from the financial news headlines. Specifically, we'll use a pre-trained language model to extract contextualized embeddings for each headline.
from transformers import AutoModel, AutoTokenizer
# Load pre-trained language model and tokenizer
model = AutoModel.from_pretrained("distilbert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
# Generate features from news headlines
headline_features = []
for headline in news_headlines["headline"]:
inputs = tokenizer(headline, return_tensors="pt")
outputs = model(**inputs)
embeddings = outputs.last_hidden_state[:, 0, :]
headline_features.append(embeddings.detach().numpy())
Building Predictive Models with AutoML
We'll use the `AutoGluon` framework to build and optimize predictive models. Specifically, we'll use the `TabularPredictor` to train a model on the combined dataset of stock prices and headline features.
from autogluon.tabular import TabularPredictor
# Combine stock prices and headline features
combined_data = pd.concat([nepse_data, pd.DataFrame(headline_features)], axis=1)
# Train predictive model
predictor = TabularPredictor(label="stock_price")
predictor.fit(combined_data)
Putting It Together
We'll now combine the code from all the steps into a single function. This function will load the data, generate features from the news headlines, build a predictive model, and evaluate its performance.
def predict_stock_price():
# Load data
nepse_data = pd.read_csv("nepse_data.csv")
news_headlines = pd.read_csv("financial_news_headlines.csv")
# Generate features from news headlines
headline_features = []
for headline in news_headlines["headline"]:
inputs = tokenizer(headline, return_tensors="pt")
outputs = model(**inputs)
embeddings = outputs.last_hidden_state[:, 0, :]
headline_features.append(embeddings.detach().numpy())
# Combine stock prices and headline features
combined_data = pd.concat([nepse_data, pd.DataFrame(headline_features)], axis=1)
# Train predictive model
predictor = TabularPredictor(label="stock_price")
predictor.fit(combined_data)
# Evaluate model performance
performance = predictor.evaluate(combined_data)
return performance
Complete Script
The full runnable script combining all steps is as follows:
#!/usr/bin/env python3
import pandas as pd
import requests
from transformers import AutoModel, AutoTokenizer
from autogluon.tabular import TabularPredictor
def predict_stock_price():
# Load data
nepse_data = pd.read_csv("nepse_data.csv")
news_headlines = pd.read_csv("financial_news_headlines.csv")
# Load pre-trained language model and tokenizer
model = AutoModel.from_pretrained("distilbert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
# Generate features from news headlines
headline_features = []
for headline in news_headlines["headline"]:
inputs = tokenizer(headline, return_tensors="pt")
outputs = model(**inputs)
embeddings = outputs.last_hidden_state[:, 0, :]
headline_features.append(embeddings.detach().numpy())
# Combine stock prices and headline features
combined_data = pd.concat([nepse_data, pd.DataFrame(headline_features)], axis=1)
# Train predictive model
predictor = TabularPredictor(label="stock_price")
predictor.fit(combined_data)
# Evaluate model performance
performance = predictor.evaluate(combined_data)
return performance
if __name__ == "__main__":
performance = predict_stock_price()
print(performance)
Expected Output
The output of the script will be the performance evaluation of the predictive model, including metrics such as accuracy, precision, recall, and F1 score.
Limitations and Tradeoffs
This approach has several limitations and tradeoffs. Firstly, the quality of the predictive model is highly dependent on the quality of the input data. Noisy or incomplete data can significantly affect the performance of the model. Secondly, the use of pre-trained language models and AutoML frameworks can be computationally expensive and require significant resources. Finally, the interpretability of the model can be limited due to the complexity of the features generated by the LLMs.
Frequently Asked Questions
What is the advantage of using LLMs for feature engineering?
The advantage of using LLMs for feature engineering is that they can automatically extract nuanced contextual features from unstructured text data, which can be difficult to capture using traditional feature engineering techniques.
How do I evaluate the performance of the predictive model?
The performance of the predictive model can be evaluated using metrics such as accuracy, precision, recall, and F1 score. These metrics can be calculated using the `evaluate` method of the `TabularPredictor` object.
Can I use this approach for other types of predictive modeling tasks?
Yes, this approach can be used for other types of predictive modeling tasks, such as classification, regression, and time series forecasting. However, the specific implementation details may vary depending on the task and the characteristics of the data.
What I'd Change
In conclusion, I believe that the integration of LLMs and AutoML frameworks has the potential to revolutionize the field of financial predictive modeling. However, there are still several challenges and limitations that need to be addressed. In future work, I would focus on improving the interpretability of the model, reducing the computational costs, and exploring the application of this approach to other domains. Additionally, I would recommend using more advanced techniques such as ensemble methods and hyperparameter tuning to further improve the performance of the model.
