As a data analyst or investor interested in the Nepali financial market, you may have encountered the challenge of accessing and analyzing financial data from Nepali sources. The data is often scattered across various websites, making it difficult to collect and normalize. In this post, we will address this problem by providing a step-by-step guide on how to scrape and normalize financial data from the Nepal Stock Exchange (NEPSE) using Python. We will cover the importance of scraping and normalizing financial data, how to use Python to scrape data from the NEPSE API, and how to normalize the data using pandas and Python.
Key Takeaways
- The importance of scraping and normalizing financial data from Nepali sources, including the NEPSE.
- How to use Python to scrape data from the NEPSE API and handle errors.
- How to normalize data using pandas and Python, including handling missing values and outliers.
The Problem
The lack of easily accessible and normalized financial data from Nepali sources, including the NEPSE, hinders the ability of analysts and investors to make informed decisions. This problem can be addressed by scraping and normalizing the data, making it more accessible and usable for analysis.
Data and Sources
The data used in this tutorial is scraped from the Nepal Stock Exchange (NEPSE) API, which provides real-time and historical data on stock prices, trading volumes, and other financial metrics. The NEPSE API documentation can be found at https://www.nepse.com.np/api/docs. The Python libraries used in this tutorial include requests, pandas, and numpy. Data accessed on 2024-09-16.
Loading the Data
To load the data, we will use the requests library to send a GET request to the NEPSE API and retrieve the data in JSON format. We will then use the pandas library to store the data in a DataFrame.
import requests
import pandas as pd
response = requests.get("https://www.nepse.com.np/api/stocks")
data = response.json()
# Convert the data to a pandas DataFrame
df = pd.DataFrame(data)
The Core Logic
The core logic of the script involves scraping the data from the NEPSE API, handling errors, and normalizing the data using pandas and Python. We will use the pandas library to handle missing values and outliers, and to perform data normalization.
def normalize_data(df):
# Handle missing values
df.fillna(0, inplace=True)
# Handle outliers
Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
df = df[~((df < (Q1 - 1.5 * IQR)) | (df > (Q3 + 1.5 * IQR))).any(axis=1)]
# Normalize the data
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
df[['close', 'high', 'low', 'open']] = scaler.fit_transform(df[['close', 'high', 'low', 'open']])
return df
Putting It Together
Now that we have loaded the data and defined the core logic, we can put the pieces together to create a complete script. We will use the requests library to load the data, the pandas library to store and normalize the data, and the numpy library to handle numerical computations.
if __name__ == "__main__":
response = requests.get("https://www.nepse.com.np/api/stocks")
data = response.json()
df = pd.DataFrame(data)
df = normalize_data(df)
print(df)
Complete Script
The full runnable script combining all steps is shown below:
#!/usr/bin/env python3
import requests
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
def normalize_data(df):
# Handle missing values
df.fillna(0, inplace=True)
# Handle outliers
Q1 = df.quantile(0.25)
Q3 = df.quantile(0.75)
IQR = Q3 - Q1
df = df[~((df < (Q1 - 1.5 * IQR)) | (df > (Q3 + 1.5 * IQR))).any(axis=1)]
# Normalize the data
scaler = MinMaxScaler()
df[['close', 'high', 'low', 'open']] = scaler.fit_transform(df[['close', 'high', 'low', 'open']])
return df
if __name__ == "__main__":
response = requests.get("https://www.nepse.com.np/api/stocks")
data = response.json()
df = pd.DataFrame(data)
df = normalize_data(df)
print(df)
Expected Output
The expected output is a pandas DataFrame containing the normalized financial data from the NEPSE API. The DataFrame should have the same columns as the original data, but with normalized values.
Limitations and Tradeoffs
This approach has several limitations and tradeoffs. The NEPSE API may have usage limits or restrictions on the amount of data that can be scraped. Additionally, the data may not be comprehensive or up-to-date, which can affect the accuracy of the analysis. Furthermore, the normalization process may not be suitable for all types of data or analysis. In a production environment, it would be necessary to handle these limitations and tradeoffs by implementing error handling, data validation, and other robustness measures.
Frequently Asked Questions
Why is it necessary to scrape and normalize financial data from Nepali sources?
Scraping and normalizing financial data from Nepali sources is necessary because the data is often scattered across various websites and may not be easily accessible or usable for analysis. By scraping and normalizing the data, analysts and investors can make more informed decisions.
How do I handle missing values and outliers in the data?
Missing values and outliers can be handled using the pandas library. Missing values can be filled using the fillna() function, and outliers can be detected and removed using the quantile() function.
What are the limitations of scraping data from the NEPSE API?
The limitations of scraping data from the NEPSE API include usage limits, restrictions on the amount of data that can be scraped, and potential errors or inconsistencies in the data.
What I'd Change
In a production environment, I would implement additional error handling and data validation measures to ensure the robustness and accuracy of the analysis. I would also consider using more advanced data normalization techniques, such as machine learning algorithms, to improve the accuracy of the analysis. Additionally, I would explore alternative data sources, such as other APIs or databases, to complement the NEPSE API and improve the comprehensiveness of the analysis.
