Mastering Advanced Anomaly Detection in Time Series Data: A Real-World Approach

Have you ever spent hours poring over a graph of server logs or financial data, trying to pinpoint that one unusual spike or dip that could signal a critical issue or opportunity? I've been in your shoes, struggling to quantify my intuition and identify anomalies that traditional thresholding methods often miss. As someone who's worked extensively with time series data, I've learned that moving beyond simple rules and leveraging advanced machine learning techniques is crucial for uncovering those elusive anomalies. In this post, we'll dive into a practical example using real Netflix stock data, exploring how to apply Isolation Forest and One-Class SVM, and how to combine their strengths for a more robust detection system.

Key Takeaways

Robust preprocessing and feature engineering are essential for capturing temporal dynamics in time series data.
Isolation Forest is an efficient and effective ensemble method for identifying anomalies by isolating sparse data points.
One-Class SVM excels at defining complex boundaries of normal behavior, making it ideal for anomaly detection in high-dimensional data.
Combining multiple algorithms can significantly improve the accuracy of anomaly detection, especially in complex, high-volume time series data.
Real-world applications require careful consideration of limitations and tradeoffs, including computational resources, data quality, and interpretability.

The Problem

In many real-world applications, traditional anomaly detection methods often fall short, missing subtle patterns or producing too many false positives. This can be due to the complexity of the data, the presence of noise or outliers, or the lack of clear boundaries between normal and anomalous behavior. As data scientists and engineers, we need to move beyond simple rules and leverage advanced machine learning techniques to uncover critical anomalies in time series data.

Data and Sources

We'll be using the Yahoo Finance API to retrieve historical stock prices for Netflix (NFLX). You can access the data directly through the Yahoo Finance website or use the yfinance library in Python. Data accessed on 2026-08-09.

Loading the Data

We'll start by loading the historical stock prices for Netflix using the yfinance library.

import yfinance as yf
data = yf.download('NFLX', start='2020-01-01', end='2022-12-31')

Preprocessing and Feature Engineering

Next, we'll preprocess the data by handling missing values, normalizing the prices, and extracting relevant features such as moving averages and standard deviations.

import pandas as pd
from sklearn.preprocessing import MinMaxScaler

data['Moving_Avg'] = data['Close'].rolling(window=7).mean()
data['Std_Dev'] = data['Close'].rolling(window=7).std()
scaler = MinMaxScaler()
data[['Close', 'Moving_Avg', 'Std_Dev']] = scaler.fit_transform(data[['Close', 'Moving_Avg', 'Std_Dev']])

Isolation Forest

We'll apply Isolation Forest to identify anomalies in the preprocessed data.

from sklearn.ensemble import IsolationForest

iforest = IsolationForest(n_estimators=100, contamination=0.01)
iforest.fit(data[['Close', 'Moving_Avg', 'Std_Dev']])
anomaly_scores = iforest.decision_function(data[['Close', 'Moving_Avg', 'Std_Dev']])

One-Class SVM

We'll apply One-Class SVM to define the complex boundaries of normal behavior in the data.

from sklearn.svm import OneClassSVM

ocsvm = OneClassSVM(kernel='rbf', gamma=0.1, nu=0.1)
ocsvm.fit(data[['Close', 'Moving_Avg', 'Std_Dev']])
anomaly_scores_ocsvm = ocsvm.decision_function(data[['Close', 'Moving_Avg', 'Std_Dev']])

Combining Multiple Algorithms

We'll combine the strengths of Isolation Forest and One-Class SVM by taking the average of their anomaly scores.

anomaly_scores_combined = (anomaly_scores + anomaly_scores_ocsvm) / 2

Complete Script

The full runnable script combining all steps:

#!/usr/bin/env python3
import yfinance as yf
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.ensemble import IsolationForest
from sklearn.svm import OneClassSVM

def load_data():
    data = yf.download('NFLX', start='2020-01-01', end='2022-12-31')
    return data

def preprocess_data(data):
    data['Moving_Avg'] = data['Close'].rolling(window=7).mean()
    data['Std_Dev'] = data['Close'].rolling(window=7).std()
    scaler = MinMaxScaler()
    data[['Close', 'Moving_Avg', 'Std_Dev']] = scaler.fit_transform(data[['Close', 'Moving_Avg', 'Std_Dev']])
    return data

def detect_anomalies(data):
    iforest = IsolationForest(n_estimators=100, contamination=0.01)
    iforest.fit(data[['Close', 'Moving_Avg', 'Std_Dev']])
    anomaly_scores = iforest.decision_function(data[['Close', 'Moving_Avg', 'Std_Dev']])

    ocsvm = OneClassSVM(kernel='rbf', gamma=0.1, nu=0.1)
    ocsvm.fit(data[['Close', 'Moving_Avg', 'Std_Dev']])
    anomaly_scores_ocsvm = ocsvm.decision_function(data[['Close', 'Moving_Avg', 'Std_Dev']])

    anomaly_scores_combined = (anomaly_scores + anomaly_scores_ocsvm) / 2
    return anomaly_scores_combined

if __name__ == "__main__":
    data = load_data()
    data = preprocess_data(data)
    anomaly_scores = detect_anomalies(data)
    print(anomaly_scores)

Expected Output

The script will output the combined anomaly scores for each data point, indicating the likelihood of an anomaly.

Limitations and Tradeoffs

This approach has several limitations and tradeoffs, including:

Computational resources: The script requires significant computational resources, especially for large datasets.
Data quality: The quality of the data can significantly impact the accuracy of the anomaly detection.
Interpretability: The combined anomaly scores can be difficult to interpret, requiring additional analysis to understand the underlying causes of the anomalies.

Frequently Asked Questions

What is the best algorithm for anomaly detection in time series data?

The best algorithm for anomaly detection in time series data depends on the specific characteristics of the data and the goals of the analysis. Isolation Forest and One-Class SVM are both effective algorithms, but they have different strengths and weaknesses.

How do I handle missing values in my time series data?

Missing values can be handled using various techniques, including interpolation, imputation, and padding. The choice of technique depends on the nature of the data and the goals of the analysis.

Can I use this approach for real-time anomaly detection?

Yes, this approach can be used for real-time anomaly detection, but it requires careful consideration of the computational resources and data quality. Additionally, the script may need to be modified to handle streaming data and real-time processing.

What I'd Change

In conclusion, combining the strengths of Isolation Forest and One-Class SVM is a powerful approach for anomaly detection in time series data. However, there are several areas for improvement, including optimizing the computational resources, improving the interpretability of the results, and integrating the approach with real-time data processing. By addressing these limitations and tradeoffs, we can create a more robust and effective anomaly detection system for complex time series data.

Next Steps: Try applying this approach to your own time series data and explore the limitations and tradeoffs in more detail. Consider optimizing the computational resources, improving the interpretability of the results, and integrating the approach with real-time data processing.

Py Data