As a data scientist or machine learning engineer, you're likely no stranger to the challenge of labeling large datasets, which can be time-consuming and expensive. In our previous post on turbocharging ML inference, we explored strategies for reducing latency in machine learning models, but what about the costs associated with labeling the data itself? This post addresses the pain point of reducing labeling costs while maintaining model performance, targeting working developers and data scientists who have already explored basic machine learning concepts and are looking for advanced strategies to optimize their workflows.
Key Takeaways
- Uncertainty sampling can be used to select the most informative samples for labeling, reducing labeling costs.
- Active learning can be integrated with model training to maintain model performance while reducing labeling costs.
- The Open Library Search API can be used to demonstrate the active learning process with real-world data.
The Problem
In many machine learning applications, the cost of labeling data can be prohibitively expensive, making it difficult to collect large, high-quality datasets. This can lead to models that are biased, inaccurate, or unable to generalize well to new data. By implementing uncertainty sampling for active learning, we can reduce the number of samples that need to be labeled, reducing the overall cost of model development.
Data and Sources
The Open Library Search API (https://openlibrary.org/search.json?q=data+science&limit=3) will be used to demonstrate the active learning process. Data accessed on 2026-07-25. This API provides a stream of unlabeled data points (book search results) that need to be classified.
Loading the Data
The first step is to load the data from the Open Library Search API. We'll use the `requests` library to send a GET request to the API and retrieve the data in JSON format.
import requests
response = requests.get("https://openlibrary.org/search.json?q=data+science&limit=3")
data = response.json()
The Core Logic
The core logic of the script involves implementing an active learning loop using uncertainty sampling. We'll use a simple uncertainty metric, such as the probability of the most likely class, to select the most informative samples for labeling.
import numpy as np
from sklearn.metrics import accuracy_score
def uncertainty_sampling(data, model):
uncertainties = []
for sample in data:
# Calculate the uncertainty of each sample
probabilities = model.predict_proba(sample)
uncertainty = 1 - np.max(probabilities)
uncertainties.append(uncertainty)
# Select the most informative samples for labeling
indices = np.argsort(uncertainties)[::-1]
return indices
Putting It Together
Now that we have the core logic implemented, we can put everything together into a single script. We'll load the data, implement the active learning loop, and train a machine learning model using the labeled samples.
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
def main():
data = load_data()
model = LogisticRegression()
labeled_samples = []
for i in range(10):
# Select the most informative samples for labeling
indices = uncertainty_sampling(data, model)
# Label the selected samples
labeled_samples.extend([data[i] for i in indices[:10]])
# Train the model using the labeled samples
X_train, X_test, y_train, y_test = train_test_split(labeled_samples, [1]*len(labeled_samples), test_size=0.2)
model.fit(X_train, y_train)
# Evaluate the model
accuracy = accuracy_score(y_test, model.predict(X_test))
print("Accuracy:", accuracy)
if __name__ == "__main__":
main()
Complete Script
The full runnable script combining all steps:
#!/usr/bin/env python3
import requests
import numpy as np
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
def load_data():
response = requests.get("https://openlibrary.org/search.json?q=data+science&limit=3")
data = response.json()
return data
def uncertainty_sampling(data, model):
uncertainties = []
for sample in data:
# Calculate the uncertainty of each sample
probabilities = model.predict_proba(sample)
uncertainty = 1 - np.max(probabilities)
uncertainties.append(uncertainty)
# Select the most informative samples for labeling
indices = np.argsort(uncertainties)[::-1]
return indices
def main():
data = load_data()
model = LogisticRegression()
labeled_samples = []
for i in range(10):
# Select the most informative samples for labeling
indices = uncertainty_sampling(data, model)
# Label the selected samples
labeled_samples.extend([data[i] for i in indices[:10]])
# Train the model using the labeled samples
X_train, X_test, y_train, y_test = train_test_split(labeled_samples, [1]*len(labeled_samples), test_size=0.2)
model.fit(X_train, y_train)
# Evaluate the model
accuracy = accuracy_score(y_test, model.predict(X_test))
print("Accuracy:", accuracy)
if __name__ == "__main__":
main()
Expected Output
The script will output the accuracy of the model at each iteration of the active learning loop.
Limitations and Tradeoffs
This approach assumes that the uncertainty metric used is effective in selecting the most informative samples for labeling. However, this may not always be the case, and other metrics may be more effective in certain scenarios. Additionally, the script uses a simple logistic regression model, which may not be suitable for all types of data.
Frequently Asked Questions
What is uncertainty sampling?
Uncertainty sampling is a technique used in active learning to select the most informative samples for labeling. It involves calculating the uncertainty of each sample and selecting the samples with the highest uncertainty.
How does active learning work?
Active learning involves iteratively selecting the most informative samples for labeling and training a machine learning model using the labeled samples. The goal is to reduce the number of samples that need to be labeled while maintaining model performance.
What are the benefits of using uncertainty sampling for active learning?
The benefits of using uncertainty sampling for active learning include reducing the number of samples that need to be labeled, improving model performance, and reducing the cost of model development.
What I'd Change
In a real-world scenario, I would use a more robust uncertainty metric, such as Bayesian neural networks or Monte Carlo dropout, to select the most informative samples for labeling. Additionally, I would use a more advanced machine learning model, such as a deep neural network, to improve model performance. I would also consider using transfer learning or pre-trained models to reduce the amount of labeled data required.
