SVR Model

Overview

Support Vector Regression (SVR) is a supervised learning model that uses the principles of Support Vector Machines (SVMs) to perform regression analysis. Unlike traditional regression models that aim to minimize the squared error, SVR aims to find a function that deviates from the true targets by a margin $\epsilon$ (epsilon) at most, while being as flat (simple) as possible. This "epsilon-insensitive" loss function makes SVR robust to outliers. SVR can be applied to time series forecasting by transforming the time series problem into a supervised learning problem through **feature engineering**.

Architecture & Components

SVR's architecture is based on finding an optimal hyperplane in a high-dimensional feature space:

• Kernel Function: SVR maps the input data into a high-dimensional feature space using a kernel function (e.g., Radial Basis Function (RBF), linear, polynomial, sigmoid). This allows SVR to model non-linear relationships in the original input space by performing linear regression in the transformed high-dimensional space.

  $ f(x) = \langle w, \Phi(x) \rangle + b $

  Where $\Phi(x)$ is the non-linear transformation to the high-dimensional feature space.
• $\epsilon$-Insensitive Loss Function: SVR introduces an $\epsilon$-tube around the regression line. Errors within this tube are not penalized, making the model robust to small errors and outliers. The objective is to minimize the error outside this $\epsilon$-tube.

  $ \text{minimize } \frac{1}{2} ||w||^2 + C \sum (\xi_i + \xi_i^*) $
  $ \text{subject to } y_i - \langle w, \Phi(x_i) \rangle - b \le \epsilon + \xi_i $
  $ \langle w, \Phi(x_i) \rangle + b - y_i \le \epsilon + \xi_i^* $
  $ \xi_i, \xi_i^* \ge 0 $

  Where $C$ is a regularization parameter that controls the trade-off between model flatness and error tolerance, and $\xi_i, \xi_i^*$ are slack variables for errors outside the $\epsilon$-tube.
• Support Vectors: Only a subset of the training data points, called "support vectors," influence the final model. These are the points that lie on or outside the $\epsilon$-tube.
• Feature Engineering: For time series forecasting, SVR relies on manually engineered features to capture temporal patterns. These typically include:
  • Lagged Features: Past values of the time series itself. [18]
  • Time-Based Features: Day of week, month, year, etc.
  • Rolling Statistics: Moving averages, standard deviations.
  • Decomposition: Separating trend, seasonality, and random effects to increase forecast accuracy. [19]

Conceptual diagram of SVR's $\epsilon$-insensitive tube and hyperplane for time series.

When to Use SVR

SVR can be a suitable choice for time series forecasting when:

• Non-linear relationships are present: Its use of kernel functions allows it to model complex non-linear patterns.
• Robustness to outliers is important: The $\epsilon$-insensitive loss function makes it less sensitive to extreme values.
• The dataset size is moderate: SVR can be computationally intensive for very large datasets.
• You are comfortable with feature engineering: It requires transforming the time series into a supervised learning problem.
• High-dimensional feature spaces are involved: It performs well in such spaces.
• You need a model that avoids overfitting by adopting structural risk minimization. [19]

Pros and Cons

Example Implementation

Here's an example of implementing SVR for time series forecasting in Python using `scikit-learn`. The process involves creating lagged features to transform the time series into a supervised learning problem, scaling the data, and then training an `SVR` model.

Python Example (using `scikit-learn` library)

import pandas as pd
import numpy as np
from sklearn.svm import SVR
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_absolute_error, mean_squared_error
import matplotlib.pyplot as plt

# 1. Create sample time series data
date_range = pd.date_range(start='2020-01-01', periods=300, freq='D')
# Simulate data with trend, seasonality, and noise
values = (100 + np.arange(300) * 0.5 + # Trend
          20 * np.sin(np.arange(300) * 2 * np.pi / 30) + # Monthly seasonality
          np.random.randn(300) * 5) # Noise
df = pd.DataFrame({'date': date_range, 'value': values})

# 2. Feature Engineering (Create lagged features) [18]
def create_lagged_features(df, lag_steps):
    for i in range(1, lag_steps + 1):
        df[f'lag_{i}'] = df['value'].shift(i)
    return df

df = create_lagged_features(df.copy(), 7) # Use last 7 days as features

# Drop rows with NaN values created by lagging
df = df.dropna()

# 3. Splitting Data (Chronological Split)
split_date = '2020-09-01'
train = df[df['date'] < split_date]
test = df[df['date'] >= split_date]

features = [col for col in df.columns if col not in ['date', 'value']]
target = 'value'

X_train, y_train = train[features], train[target]
X_test, y_test = test[features], test[target]

# 4. Create a pipeline with scaling and SVR [18]
# StandardScaler is often crucial for SVR
# kernel: 'rbf' (Radial Basis Function) is common for non-linear data
# C: Regularization parameter. Higher C means less regularization.
# epsilon: Epsilon-tube parameter. Errors within this margin are ignored.
model_svr = make_pipeline(StandardScaler(), SVR(kernel='rbf', C=1.0, epsilon=0.1))

# 5. Fit the SVR model
model_svr.fit(X_train, y_train)

# 6. Make Predictions
predictions = model_svr.predict(X_test)

# 7. Evaluate Model Performance
mae = mean_absolute_error(y_test, predictions)
rmse = np.sqrt(mean_squared_error(y_test, predictions))
print(f"MAE: {mae:.3f}")
print(f"RMSE: {rmse:.3f}")

# 8. Plotting Results
plt.figure(figsize=(14, 7))
plt.plot(train['date'], train['value'], label='Training Data', color='blue')
plt.plot(test['date'], y_test, label='Actual Test Data', color='orange')
plt.plot(test['date'], predictions, label='SVR Predictions', color='green', linestyle='--')
plt.title('SVR Time Series Forecasting')
plt.xlabel('Date')
plt.ylabel('Value')
plt.legend()
plt.grid(True)
plt.show()
                        

Dependencies & Resources

Dependencies: pandas, numpy, scikit-learn, matplotlib (for plotting).

• Scikit-learn SVR Documentation ↗
• Time series forecasting with SVR in scikit learn (Stack Overflow) ↗ [18]
• Localized support vector regression for time series prediction (ResearchGate) ↗ [20]
• Support Vector Regression for Time Series Forecasting (Blog) ↗ [21]
• Support Vector Regression for Forecasting Cases (PDF) ↗ [22]
• Time series forecasting by a seasonal support vector regression model (ResearchGate) ↗ [19]