As a Python developer or data scientist, understanding the trends and patterns of PyPI package downloads is crucial for informing development decisions, identifying emerging trends, and optimizing package maintenance. However, analyzing and predicting package popularity can be a daunting task, especially when dealing with large datasets. In this post, we'll explore how to build a predictive model using scikit-learn and the PyPI Stats API to forecast PyPI package download trends.
Key Takeaways
- How to fetch PyPI package download data from the PyPI Stats API
- How to build a predictive model using scikit-learn's LinearRegression and RandomForestRegressor algorithms
- How to evaluate the performance of the trained models using metrics such as mean squared error and R-squared
The Problem
The problem of analyzing and predicting PyPI package download trends is a real-world challenge that many developers and data scientists face. With the increasing popularity of Python and the growing number of packages available on PyPI, understanding package trends is more important than ever.
Data and Sources
The PyPI Stats API provides daily download counts for PyPI packages. We'll be using the https://pypistats.org/api/packages/requests/overall endpoint to fetch the daily download counts for the `requests` package. Data accessed on 2026-07-20.
Loading the Data
To fetch the data, we'll use the `requests` library to send a GET request to the PyPI Stats API.
import requests
response = requests.get("https://pypistats.org/api/packages/requests/overall")
data = response.json()
Data Preprocessing
After fetching the data, we'll need to preprocess it to prepare it for modeling. We'll use pandas to convert the data into a DataFrame and perform any necessary cleaning and feature engineering.
import pandas as pd
df = pd.DataFrame(data)
df['date'] = pd.to_datetime(df['date'])
df.set_index('date', inplace=True)
Model Building
Next, we'll build a predictive model using scikit-learn's LinearRegression and RandomForestRegressor algorithms. We'll split the data into training and testing sets and use the training set to train the models.
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
X = df.index.map(pd.to_datetime).astype(int).values.reshape(-1, 1)
y = df['downloads'].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
lr_model = LinearRegression()
lr_model.fit(X_train, y_train)
rf_model = RandomForestRegressor()
rf_model.fit(X_train, y_train)
Model Evaluation
After training the models, we'll evaluate their performance using metrics such as mean squared error and R-squared.
from sklearn.metrics import mean_squared_error, r2_score
lr_y_pred = lr_model.predict(X_test)
rf_y_pred = rf_model.predict(X_test)
lr_mse = mean_squared_error(y_test, lr_y_pred)
rf_mse = mean_squared_error(y_test, rf_y_pred)
lr_r2 = r2_score(y_test, lr_y_pred)
rf_r2 = r2_score(y_test, rf_y_pred)
Putting It Together
Finally, we'll put all the pieces together into a single function that fetches the data, preprocesses it, builds the models, and evaluates their performance.
def analyze_pypi_trends():
# fetch data
response = requests.get("https://pypistats.org/api/packages/requests/overall")
data = response.json()
# preprocess data
df = pd.DataFrame(data)
df['date'] = pd.to_datetime(df['date'])
df.set_index('date', inplace=True)
# build models
X = df.index.map(pd.to_datetime).astype(int).values.reshape(-1, 1)
y = df['downloads'].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
lr_model = LinearRegression()
lr_model.fit(X_train, y_train)
rf_model = RandomForestRegressor()
rf_model.fit(X_train, y_train)
# evaluate models
lr_y_pred = lr_model.predict(X_test)
rf_y_pred = rf_model.predict(X_test)
lr_mse = mean_squared_error(y_test, lr_y_pred)
rf_mse = mean_squared_error(y_test, rf_y_pred)
lr_r2 = r2_score(y_test, lr_y_pred)
rf_r2 = r2_score(y_test, rf_y_pred)
print(f"Linear Regression MSE: {lr_mse}, R2: {lr_r2}")
print(f"Random Forest Regressor MSE: {rf_mse}, R2: {rf_r2}")
if __name__ == "__main__":
analyze_pypi_trends()
Complete Script
The full runnable script combining all steps:
#!/usr/bin/env python3
import requests
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score
def analyze_pypi_trends():
# fetch data
response = requests.get("https://pypistats.org/api/packages/requests/overall")
data = response.json()
# preprocess data
df = pd.DataFrame(data)
df['date'] = pd.to_datetime(df['date'])
df.set_index('date', inplace=True)
# build models
X = df.index.map(pd.to_datetime).astype(int).values.reshape(-1, 1)
y = df['downloads'].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
lr_model = LinearRegression()
lr_model.fit(X_train, y_train)
rf_model = RandomForestRegressor()
rf_model.fit(X_train, y_train)
# evaluate models
lr_y_pred = lr_model.predict(X_test)
rf_y_pred = rf_model.predict(X_test)
lr_mse = mean_squared_error(y_test, lr_y_pred)
rf_mse = mean_squared_error(y_test, rf_y_pred)
lr_r2 = r2_score(y_test, lr_y_pred)
rf_r2 = r2_score(y_test, rf_y_pred)
print(f"Linear Regression MSE: {lr_mse}, R2: {lr_r2}")
print(f"Random Forest Regressor MSE: {rf_mse}, R2: {rf_r2}")
if __name__ == "__main__":
analyze_pypi_trends()
Expected Output
The script will output the mean squared error and R-squared values for both the Linear Regression and Random Forest Regressor models.
Limitations and Tradeoffs
This approach has several limitations and tradeoffs. First, the PyPI Stats API only provides daily download counts, which may not be sufficient for real-time analysis. Second, the models are trained on a limited dataset, which may not generalize well to other packages or time periods. Finally, the models are relatively simple and may not capture complex relationships between package downloads and other factors.
Frequently Asked Questions
What is the PyPI Stats API?
The PyPI Stats API is a service that provides download statistics for PyPI packages.
How do I fetch data from the PyPI Stats API?
You can fetch data from the PyPI Stats API using the `requests` library in Python.
What is the difference between Linear Regression and Random Forest Regressor?
Linear Regression is a linear model that assumes a linear relationship between the independent and dependent variables, while Random Forest Regressor is an ensemble model that combines multiple decision trees to predict the dependent variable.
What I'd Change
In a real-world scenario, I would consider using more advanced models, such as LSTM or Prophet, to capture complex relationships between package downloads and other factors. I would also consider using more data, such as historical download counts, to improve the accuracy of the models. Additionally, I would consider using techniques such as feature engineering and hyperparameter tuning to further improve the performance of the models.
