← Back to Model Library

LSTM Model

Long Short-Term Memory Network

Overview

Long Short-Term Memory (LSTM) networks are a specialized type of Recurrent Neural Network (RNN) designed to overcome the limitations of traditional RNNs in learning long-term dependencies. Standard RNNs suffer from the vanishing gradient problem, which makes it difficult for them to remember information over long sequences. LSTMs introduce a "memory cell" and gating mechanisms to control the flow of information, allowing them to selectively remember or forget information over extended periods.

Architecture & Components

The core innovation of the LSTM is the memory cell ($C_t$), which acts as a conveyor belt of information through the sequence. The flow of information into and out of this cell is regulated by three main "gates":

  • Forget Gate ($f_t$): This gate decides what information to discard from the cell state. It looks at the previous hidden state ($h_{t-1}$) and the current input ($x_t$) and outputs a number between 0 and 1 for each number in the previous cell state $C_{t-1}$. A 1 means "completely keep this" while a 0 means "completely get rid of this."
  • Input Gate ($i_t$): This gate decides which new information to store in the cell state. It consists of two parts: a sigmoid layer that decides which values to update, and a tanh layer that creates a vector of new candidate values, $\tilde{C}_t$, that could be added to the state.
  • Output Gate ($o_t$): This gate determines the next hidden state, which is a filtered version of the cell state. First, a sigmoid layer decides which parts of the cell state we’re going to output. Then, we put the cell state through a tanh function (to push the values to be between -1 and 1) and multiply it by the output of the sigmoid gate.
LSTM Cell Diagram

Conceptual diagram of an LSTM cell showing the forget, input, and output gates controlling the cell state.

When to Use LSTM

LSTMs are a powerful choice for complex time series problems, especially when:

  • The data contains complex, non-linear patterns that classical models cannot capture.
  • There are long-term dependencies in the data (e.g., the current value depends on events that happened many time steps ago).
  • You have a large amount of training data. Deep learning models like LSTMs typically require more data than classical statistical models.
  • Performance is more critical than model interpretability. While techniques exist to interpret LSTMs, they are inherently more of a "black box" than models like ARIMA.

Example Implementation

Here's a conceptual implementation of an LSTM model for time series forecasting using TensorFlow/Keras. The process involves reshaping the data into sequences of inputs and corresponding outputs, scaling the data, and then building and training the network.

# Import necessary libraries
import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense
from sklearn.preprocessing import MinMaxScaler

# Generate sample data with non-linear patterns
np.random.seed(42)
n_samples = 200
time = np.arange(n_samples)
data = np.sin(time / 20) * 10 + time * 0.1 + np.random.randn(n_samples) * 2
data = data.reshape(-1, 1)

# Scale data to be between 0 and 1
scaler = MinMaxScaler(feature_range=(0, 1))
scaled_data = scaler.fit_transform(data)

# Create sequences for LSTM
def create_dataset(dataset, look_back=1):
    X, Y = [], []
    for i in range(len(dataset) - look_back - 1):
        a = dataset[i:(i + look_back), 0]
        X.append(a)
        Y.append(dataset[i + look_back, 0])
    return np.array(X), np.array(Y)

look_back = 10
X, y = create_dataset(scaled_data, look_back)

# Reshape input to be [samples, time steps, features]
X = np.reshape(X, (X.shape[0], X.shape[1], 1))

# Build the LSTM model
model = Sequential()
model.add(LSTM(50, return_sequences=True, input_shape=(look_back, 1)))
model.add(LSTM(50))
model.add(Dense(1))

model.compile(optimizer='adam', loss='mean_squared_error')

# Train the model
model.fit(X, y, epochs=20, batch_size=1, verbose=2)

# For prediction, you would take the last 'look_back' points from the training data,
# predict the next point, append it, and repeat.
print("LSTM model training complete.")