ARIMA-LSTM Hybrid Model

Overview

The ARIMA-LSTM hybrid model combines the strengths of traditional statistical time series methods with the advanced capabilities of deep learning. This dual-stage forecasting process leverages ARIMA (AutoRegressive Integrated Moving Average) to capture the linear patterns (trend and seasonality) in a time series, and then uses a Long Short-Term Memory (LSTM) neural network to model the complex non-linear relationships found in the residuals (errors) of the ARIMA model. This hybridization aims to achieve superior forecasting performance by addressing both linear and non-linear components that often coexist in real-world time series data.

Architecture & Components

The ARIMA-LSTM hybrid model typically follows a two-stage sequential process:

1. Stage 1: ARIMA Modeling (Linear Component)

   A classical ARIMA model is first applied to the raw time series data. The ARIMA component is responsible for capturing and forecasting the transparent linear trends and seasonal patterns. It identifies the autoregressive (AR), integrated (I), and moving average (MA) orders (p, d, q) that best describe the linear dependencies in the data. After fitting, the ARIMA model generates in-sample predictions, and the **residuals** (the differences between the actual values and the ARIMA's fitted values) are calculated. These residuals are assumed to primarily contain the non-linear patterns that the ARIMA model could not capture.

   $ R_t = Y_t - \hat{Y}_t^{\text{ARIMA}} $

   Where $R_t$ are the residuals, $Y_t$ is the actual value, and $\hat{Y}_t^{\text{ARIMA}}$ is the ARIMA's fitted value.
2. Stage 2: LSTM Modeling (Non-linear Residuals)

   An LSTM neural network is then trained on these residuals. The LSTM's ability to learn complex non-linear relationships and long-term dependencies makes it ideal for modeling the intricate patterns that remain after the linear component has been accounted for. The LSTM takes past residuals as input and learns a function to forecast the future deviation of the linear predictions.

   $ \hat{R}_t^{\text{LSTM}} = \text{LSTM}(R_{t-1}, R_{t-2}, \dots, R_{t-w}) $

   Where $\hat{R}_t^{\text{LSTM}}$ is the LSTM's forecast of the residual, and $w$ is the look-back window for the LSTM.
3. Final Forecast Combination:

   The final forecast is obtained by summing the forecasts from both components: the linear forecast from ARIMA and the non-linear residual forecast from LSTM.

   $ \hat{Y}_t^{\text{Hybrid}} = \hat{Y}_t^{\text{ARIMA}} + \hat{R}_t^{\text{LSTM}} $

Conceptual diagram of the ARIMA-LSTM hybrid model, showing sequential processing.

When to Use ARIMA-LSTM Hybrid

The ARIMA-LSTM hybrid model is particularly effective for:

• Time series with both linear and non-linear patterns: This is common in real-world data where underlying processes might have both predictable linear trends/seasonalities and complex, non-linear dynamics.
• Achieving high forecasting accuracy: By combining complementary strengths, it often outperforms standalone ARIMA or LSTM models.
• Short-horizon and long-horizon forecasts: Hybrid methods have shown consistent outperformance across various forecasting horizons.
• When interpretability of the linear component is desired: The ARIMA part provides a transparent baseline.
• As a robust solution for challenging time series data.

Pros and Cons

Example Implementation

Implementing an ARIMA-LSTM hybrid model involves several steps: fitting ARIMA, extracting residuals, preparing residuals for LSTM, training LSTM, and combining forecasts. Here's a conceptual Python example demonstrating this process.

Python Example (Conceptual)

                        
import pandas as pd
import numpy as np
from statsmodels.tsa.arima.model import ARIMA
from sklearn.preprocessing import MinMaxScaler
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense
from sklearn.metrics import mean_absolute_error, mean_squared_error
import matplotlib.pyplot as plt

# 1. Generate sample data with both linear trend/seasonality and some non-linearity
np.random.seed(42)
n_samples = 200
time_idx = np.arange(n_samples)
# Linear trend + seasonality
linear_component = 50 + 0.5 * time_idx + 10 * np.sin(time_idx * 2 * np.pi / 30)
# Add some non-linear, autoregressive-like noise
non_linear_noise = np.zeros(n_samples)
for i in range(1, n_samples):
    non_linear_noise[i] = 0.3 * non_linear_noise[i-1] + np.random.normal(0, 1) * (1 + np.sin(i/50))

original_series = linear_component + non_linear_noise
series = pd.Series(original_series, index=pd.date_range(start='2020-01-01', periods=n_samples, freq='D'))

# 2. Split data into train and test sets (chronological)
train_size = 150
train_series, test_series = series[0:train_size], series[train_size:n_samples]

# --- Stage 1: ARIMA Modeling ---
# 3. Fit ARIMA model to capture linear patterns
# (p,d,q) orders need to be determined via ACF/PACF or auto_arima
# For demonstration, let's assume (5,1,0) for non-seasonal data
arima_order = (5,1,0)
arima_model = ARIMA(train_series, order=arima_order)
arima_model_fit = arima_model.fit()

# 4. Get ARIMA in-sample predictions and residuals
arima_train_pred = arima_model_fit.predict(start=0, end=len(train_series)-1)
arima_residuals = train_series - arima_train_pred

print("ARIMA Model Summary:")
print(arima_model_fit.summary())
print(f"\nARIMA Residuals (first 5): {arima_residuals.head().values}")

# --- Stage 2: LSTM Modeling on Residuals ---
# 5. Prepare residuals for LSTM (supervised learning format)
# LSTM needs sequences as input, so we create lagged features from residuals
look_back = 10 # Number of past residuals to use as input for LSTM
scaler = MinMaxScaler(feature_range=(0, 1))
scaled_residuals = scaler.fit_transform(arima_residuals.values.reshape(-1, 1))

def create_lstm_dataset(dataset, look_back=1):
    X, Y =,
    for i in range(len(dataset) - look_back):
        X.append(dataset[i:(i + look_back), 0])
        Y.append(dataset[i + look_back, 0])
    return np.array(X), np.array(Y)

X_residuals, y_residuals = create_lstm_dataset(scaled_residuals, look_back)

# Reshape input to be [samples, time steps, features] for LSTM
X_residuals = np.reshape(X_residuals, (X_residuals.shape, X_residuals.shape[1], 1))

# 6. Build and train LSTM model on residuals
lstm_model = Sequential()
lstm_model.add(LSTM(50, activation='relu', input_shape=(look_back, 1)))
lstm_model.add(Dense(1))
lstm_model.compile(optimizer='adam', loss='mean_squared_error')

print("\nStarting LSTM training on ARIMA residuals...")
lstm_model.fit(X_residuals, y_residuals, epochs=50, batch_size=1, verbose=0) # Reduced epochs for demo
print("LSTM training complete.")

# --- Forecasting and Combination ---
# 7. Make multi-step forecasts
forecast_steps = len(test_series)

# ARIMA forecast for the future
arima_forecast_future = arima_model_fit.forecast(steps=forecast_steps)

# LSTM forecast for future residuals (recursive prediction)
# Start with the last 'look_back' residuals from training
last_residuals_sequence = scaled_residuals[-look_back:]
lstm_future_residuals_scaled =
current_lstm_input = last_residuals_sequence.reshape(1, look_back, 1)

for _ in range(forecast_steps):
    next_residual_pred_scaled = lstm_model.predict(current_lstm_input, verbose=0)
    lstm_future_residuals_scaled.append(next_residual_pred_scaled)
    # Update input sequence: remove oldest, add new prediction
    current_lstm_input = np.append(current_lstm_input[:, 1:, :], [[[next_residual_pred_scaled]]], axis=1)

lstm_future_residuals = scaler.inverse_transform(np.array(lstm_future_residuals_scaled).reshape(-1, 1)).flatten()

# 8. Combine forecasts
hybrid_forecast = arima_forecast_future.values + lstm_future_residuals

# 9. Evaluate Hybrid Model
mae = mean_absolute_error(test_series, hybrid_forecast)
rmse = np.sqrt(mean_squared_error(test_series, hybrid_forecast))
print(f"\nHybrid Model MAE: {mae:.3f}")
print(f"Hybrid Model RMSE: {rmse:.3f}")

# 10. Plotting Results
plt.figure(figsize=(14, 7))
plt.plot(train_series.index, train_series, label='Training Data', color='blue')
plt.plot(test_series.index, test_series, label='Actual Test Data', color='orange')
plt.plot(test_series.index, hybrid_forecast, label='ARIMA-LSTM Hybrid Forecast', color='green', linestyle='--')
plt.title('ARIMA-LSTM Hybrid Time Series Forecasting')
plt.xlabel('Date')
plt.ylabel('Value')
plt.legend()
plt.grid(True)
plt.show()