← Back to Model Library

FEDformer Model

Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting

Overview

FEDformer (Frequency Enhanced Decomposed Transformer) is a Transformer-based model designed for long-term time series forecasting. It addresses the limitations of standard Transformers, which are computationally expensive and struggle to capture the global view of time series. FEDformer combines the Transformer architecture with seasonal-trend decomposition and integrates Fourier-based decomposition to process time series in the frequency domain. This approach allows it to capture global patterns and long-range dependencies more effectively while significantly reducing computational overhead.

Architecture & Components

FEDformer's architecture is built on the Transformer framework but introduces key innovations:

  • Seasonal-Trend Decomposition: Similar to Autoformer, FEDformer first decomposes the time series input into a trend and a seasonal component. The trend captures long-term growth or decline, while the seasonal component captures recurring patterns. This decomposition helps the model focus on distinct aspects of the time series.
  • Frequency-Domain Processing (Fourier-based Decomposition): This is a core innovation. Instead of processing the entire time series in the time domain, FEDformer converts parts of the time series (specifically the seasonal component) into the frequency domain using Fourier transforms. This allows the model to detect dominant patterns by focusing on the strongest frequencies and effectively filtering out noise.
  • Frequency Enhanced Block (FEB) and Frequency Enhanced Attention (FEA): These are the mechanisms through which Fourier analysis is integrated into the neural network. FEB and FEA operate in the frequency domain, where attention mechanisms are applied to identify the most meaningful signals. This leverages the sparsity and low-rank characteristics of time series in the Fourier domain, leading to a more compact and informative representation.
  • Linear Complexity: By exploiting frequency-domain sparsity and random selection of frequency bases, FEDformer achieves a linear complexity with respect to the sequence length, making it more efficient than standard Transformers.
  • Encoder-Decoder Structure: FEDformer maintains an encoder-decoder structure. The encoder processes the input, and the decoder generates the forecast, with the final prediction being the sum of refined decomposed components.
FEDformer Architecture Diagram

Conceptual diagram of FEDformer's architecture with frequency-domain processing.

When to Use FEDformer

FEDformer is particularly suitable for:

  • Long-term time series forecasting where efficiency and accuracy are critical.
  • Data that exhibits clear periodic patterns, as it leverages Fourier transforms to identify dominant frequencies.
  • Applications where noise filtering is important, as it focuses on strong frequencies to highlight key signals.
  • When you need a Transformer-based model that is more computationally efficient than vanilla Transformers for long sequences.

Pros and Cons

Pros

  • Reduced Complexity: Achieves linear complexity ($O(L)$) with sequence length, making it highly efficient for long time series.
  • Effective for Periodic Data: Leverages Fourier-based decomposition to effectively capture and model periodic patterns.
  • Noise Filtering: By focusing on dominant frequencies, it effectively filters out noise and highlights important signals.
  • Improved Accuracy: Demonstrates state-of-the-art forecasting accuracy, reducing prediction error significantly compared to other methods.
  • Captures Global View: Its decomposition and frequency analysis help capture overall trends and global patterns.

Cons

  • Potential for Underfitting: Only handling low-frequency components might lead to underfitting if high-frequency details are crucial.
  • Limited Flexibility for Irregular Data: Reliance on frequency-domain transforms may limit its flexibility in handling irregular or highly non-stationary time series.
  • Outperformed by Some Models: While strong, other models like TSAA have shown to consistently outperform it across various error metrics in some benchmarks.
  • Less Interpretable: Frequency-domain operations can make the model less interpretable than simpler decomposition methods.

Example Implementation

FEDformer is primarily implemented in PyTorch, with the official code provided by MAZiqing/FEDformer. It is also included in the THUML/Autoformer repository. The typical usage involves running bash scripts for specific datasets.

PyTorch Example (using MAZiqing/FEDformer)

                        
# 1. Clone the official FEDformer repository
# git clone https://github.com/MAZiqing/FEDformer.git
# cd FEDformer

# 2. Install Python >= 3.8 and PyTorch 1.9.0 (or compatible versions)
# pip install -r requirements.txt # (assuming a requirements.txt exists or install manually)

# 3. Download datasets
# Datasets can be obtained from Autoformer or Informer repositories (links in FEDformer README).
# Download them and place them in a './dataset' folder in the root of the cloned repository.

# 4. Run a training script for a specific dataset (e.g., ETTm1 for univariate forecasting)
# These scripts are located in the './scripts' directory.
echo "Running FEDformer training script for ETTm1 dataset (univariate)..."
bash./scripts/ETT_script/FEDformer_ETTm1_univariate.sh

# This script will typically:
# - Set up model parameters (e.g., sequence length, prediction length, number of encoder/decoder layers)
# - Load the ETTm1 dataset
# - Train the FEDformer model
# - Evaluate its performance (RMSE, MAE) and save results.

# Example of what the script might contain (simplified):
# python main_long_term_forecast.py \
#   --model FEDformer \
#   --data ETTm1 \
#   --features S \
#   --seq_len 96 \
#   --label_len 48 \
#   --pred_len 24 \
#   --e_layers 2 \
#   --d_layers 1 \
#   --factor 3 \
#   --enc_in 1 \
#   --dec_in 1 \
#   --c_out 1 \
#   --des Exp \
#   --itr 1 \
#   --train_epochs 10 \
#   --batch_size 32 \
#   --learning_rate 0.0001 \
#   --root_path./dataset/ETT-small/ \
#   --data_path ETTm1.csv \
#   --checkpoints./checkpoints/

echo "FEDformer training script executed. Check output directory for results."

# For multivariate forecasting, you would run a similar script with --features M
# bash./scripts/ETT_script/FEDformer_ETTm1_multivariate.sh

Dependencies & Resources

Dependencies: pandas, numpy, torch, matplotlib (for plotting).