ARIMA-Transformer Hybrid Model

Overview

The ARIMA-Transformer hybrid model combines the strengths of traditional statistical time series methods with the advanced capabilities of Transformer networks. This dual-stage forecasting process leverages ARIMA (AutoRegressive Integrated Moving Average) to capture the linear patterns (trend and seasonality) in a time series, and then uses a Transformer model to capture the complex non-linear relationships and long-range dependencies found in the residuals (errors) of the ARIMA model. This hybridization aims to achieve superior forecasting performance by addressing both linear and non-linear components that often coexist in real-world time series data, particularly benefiting from the Transformer's ability to model intricate sequential patterns.

Architecture & Components

The ARIMA-Transformer hybrid model typically follows a two-stage sequential process:

1. Stage 1: ARIMA Modeling (Linear Component)

   A classical ARIMA model is first applied to the raw time series data. The ARIMA component is responsible for capturing and forecasting the transparent linear trends and seasonal patterns. After fitting, the ARIMA model generates in-sample predictions, and the **residuals** (the differences between the actual values and the ARIMA's fitted values) are calculated. These residuals are assumed to primarily contain the non-linear patterns that the ARIMA model could not capture.

   $ R_t = Y_t - \hat{Y}_t^{\text{ARIMA}} $

   Where $R_t$ are the residuals, $Y_t$ is the actual value, and $\hat{Y}_t^{\text{ARIMA}}$ is the ARIMA's fitted value.
2. Stage 2: Transformer Modeling (Non-linear Residuals)

   A Transformer model is then trained on these residuals. The Transformer's self-attention mechanism allows it to capture complex non-linear relationships and long-range dependencies in the residual series, overcoming the limitations of traditional RNNs. The Transformer takes past residuals as input and learns a function to forecast the future deviation of the linear predictions.

   $ \hat{R}_t^{\text{Transformer}} = \text{Transformer}(R_{t-w}, \dots, R_{t-1}) $

   Where $\hat{R}_t^{\text{Transformer}}$ is the Transformer's forecast of the residual, and $w$ is the look-back window for the Transformer.
3. Final Forecast Combination:

   The final forecast is obtained by summing the forecasts from both components: the linear forecast from ARIMA and the non-linear residual forecast from the Transformer.

   $ \hat{Y}_t^{\text{Hybrid}} = \hat{Y}_t^{\text{ARIMA}} + \hat{R}_t^{\text{Transformer}} $

Conceptual diagram of the ARIMA-Transformer hybrid model, showing sequential processing.

When to Use ARIMA-Transformer Hybrid

The ARIMA-Transformer hybrid model is particularly effective for:

• Time series with both clear linear patterns and complex non-linear, long-range dependencies: This is common in real-world data where underlying processes might have both predictable linear trends/seasonalities and intricate, non-linear dynamics that benefit from Transformer's attention mechanism.
• Achieving high forecasting accuracy: By combining complementary strengths, it often outperforms standalone ARIMA or Transformer models.
• Short-horizon and long-horizon forecasts: Hybrid methods have shown consistent outperformance across various forecasting horizons.
• When interpretability of the linear component is desired: The ARIMA part provides a transparent baseline.
• As a robust solution for challenging time series data.

Pros and Cons

Example Implementation

Implementing an ARIMA-Transformer hybrid model involves several steps: fitting ARIMA, extracting residuals, preparing residuals for the Transformer, training the Transformer, and combining forecasts. Here's a conceptual Python example demonstrating this process.

Python Example (Conceptual)

import pandas as pd
import numpy as np
from statsmodels.tsa.arima.model import ARIMA
from sklearn.preprocessing import MinMaxScaler
import matplotlib.pyplot as plt

# TensorFlow/Keras for Transformer
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, Dropout, LayerNormalization
from tensorflow.keras.models import Model

# 1. Generate sample data with both linear trend/seasonality and some non-linearity
np.random.seed(42)
n_samples = 200
time_idx = np.arange(n_samples)
# Linear trend + seasonality
linear_component = 50 + 0.5 * time_idx + 10 * np.sin(time_idx * 2 * np.pi / 30)
# Add some non-linear, autoregressive-like noise
non_linear_noise = np.zeros(n_samples)
for i in range(1, n_samples):
    non_linear_noise[i] = 0.3 * non_linear_noise[i-1] + np.random.normal(0, 1) * (1 + np.sin(i/50))

original_series = linear_component + non_linear_noise
series = pd.Series(original_series, index=pd.date_range(start='2020-01-01', periods=n_samples, freq='D'))

# 2. Split data into train and test sets (chronological)
train_size = 150
train_series, test_series = series[0:train_size], series[train_size:n_samples]

# --- Stage 1: ARIMA Modeling ---
# 3. Fit ARIMA model to capture linear patterns
# (p,d,q) orders need to be determined via ACF/PACF or auto_arima
arima_order = (5,1,0)
arima_model = ARIMA(train_series, order=arima_order)
arima_model_fit = arima_model.fit()

# 4. Get ARIMA in-sample predictions and residuals
arima_train_pred = arima_model_fit.predict(start=0, end=len(train_series)-1)
arima_residuals = train_series - arima_train_pred

print("ARIMA Model Summary:")
print(arima_model_fit.summary())
print(f"\nARIMA Residuals (first 5): {arima_residuals.head().values}")

# --- Stage 2: Transformer Modeling on Residuals ---
# 5. Prepare residuals for Transformer (supervised learning format)
look_back = 10 # Number of past residuals to use as input for Transformer
scaler = MinMaxScaler(feature_range=(0, 1))
scaled_residuals = scaler.fit_transform(arima_residuals.values.reshape(-1, 1))

def create_transformer_dataset(data, look_back=1):
    X, Y =,
    for i in range(len(data) - look_back):
        X.append(data[i:(i + look_back), 0])
        Y.append(data[i + look_back, 0])
    return np.array(X), np.array(Y)

X_residuals, y_residuals = create_transformer_dataset(scaled_residuals, look_back)

# Reshape input to be [samples, time steps, features] for Transformer
X_residuals = np.reshape(X_residuals, (X_residuals.shape, X_residuals.shape[1], 1))

# Positional Embedding for Transformer
class PositionalEmbedding(tf.keras.layers.Layer):
    def __init__(self, sequence_length, embed_dim, **kwargs):
        super().__init__(**kwargs)
        self.position_embeddings = tf.keras.layers.Embedding(sequence_length, embed_dim)
        self.embed_dim = embed_dim
    def call(self, inputs):
        length = tf.shape(inputs)[-2]
        positions = tf.range(start=0, limit=length, delta=1)
        return inputs + self.position_embeddings(positions)

# Transformer Block (Encoder Layer)
class TransformerBlock(tf.keras.layers.Layer):
    def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1, **kwargs):
        super().__init__(**kwargs)
        self.att = tf.keras.layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)
        self.ffn = tf.keras.Sequential()
        self.layernorm1 = LayerNormalization(epsilon=1e-6)
        self.layernorm2 = LayerNormalization(epsilon=1e-6)
        self.dropout1 = Dropout(rate)
        self.dropout2 = Dropout(rate)
    def call(self, inputs, training):
        attn_output = self.att(inputs, inputs)
        attn_output = self.dropout1(attn_output, training=training)
        out1 = self.layernorm1(inputs + attn_output)
        ffn_output = self.ffn(out1)
        ffn_output = self.dropout2(ffn_output, training=training)
        return self.layernorm2(out1 + ffn_output)

# 6. Build and train Transformer model on residuals
embed_dim = 32
num_heads = 4
ff_dim = 32
num_transformer_blocks = 2

inputs = Input(shape=(look_back, 1))
x = Dense(embed_dim)(inputs) # Project input features to embed_dim
x = PositionalEmbedding(look_back, embed_dim)(x)

for _ in range(num_transformer_blocks):
    x = TransformerBlock(embed_dim, num_heads, ff_dim)(x)

outputs = Dense(1)(x[:, -1, :]) # Predict the next single value

transformer_model = Model(inputs=inputs, outputs=outputs)
transformer_model.compile(optimizer='adam', loss='mean_squared_error')

print("\nStarting Transformer training on ARIMA residuals...")
transformer_model.fit(X_residuals, y_residuals, epochs=50, batch_size=32, verbose=0) # Reduced epochs for demo
print("Transformer training complete.")

# --- Forecasting and Combination ---
# 7. Make multi-step forecasts
forecast_steps = len(test_series)

# ARIMA forecast for the future
arima_forecast_future = arima_model_fit.forecast(steps=forecast_steps)

# Transformer forecast for future residuals (recursive prediction)
last_residuals_sequence = scaled_residuals[-look_back:]
transformer_future_residuals_scaled =
current_transformer_input = last_residuals_sequence.reshape(1, look_back, 1)

for _ in range(forecast_steps):
    next_residual_pred_scaled = transformer_model.predict(current_transformer_input, verbose=0)
    transformer_future_residuals_scaled.append(next_residual_pred_scaled)
    # Update input sequence: remove oldest, add new prediction
    current_transformer_input = np.append(current_transformer_input[:, 1:, :], [[[next_residual_pred_scaled]]], axis=1)

transformer_future_residuals = scaler.inverse_transform(np.array(transformer_future_residuals_scaled).reshape(-1, 1)).flatten()

# 8. Combine forecasts
hybrid_forecast = arima_forecast_future.values + transformer_future_residuals

# 9. Evaluate Hybrid Model
mae = mean_absolute_error(test_series, hybrid_forecast)
rmse = np.sqrt(mean_squared_error(test_series, hybrid_forecast))
print(f"\nHybrid Model MAE: {mae:.3f}")
print(f"Hybrid Model RMSE: {rmse:.3f}")

# 10. Plotting Results
plt.figure(figsize=(14, 7))
plt.plot(train_series.index, train_series, label='Training Data', color='blue')
plt.plot(test_series.index, test_series, label='Actual Test Data', color='orange')
plt.plot(test_series.index, hybrid_forecast, label='ARIMA-Transformer Hybrid Forecast', color='green', linestyle='--')
plt.title('ARIMA-Transformer Hybrid Time Series Forecasting')
plt.xlabel('Date')
plt.ylabel('Value')
plt.legend()
plt.grid(True)
plt.show()
                    

Dependencies & Resources

Dependencies: pandas, numpy, statsmodels, tensorflow/keras, scikit-learn, matplotlib (for plotting).

• A Hybrid Framework Integrating Traditional Models and Deep Learning for Multi-Scale Time Series Forecasting (MDPI Paper) ↗
• A Hybrid Forecasting Model Based on ARIMA and Transformer (ResearchGate) ↗
• Hybrid-Time-Series-Modeling GitHub Repository (includes conceptual hybrid frameworks) ↗