Chapter 2: Machine Learning Models

Overview of ML in Trading

Machine learning has revolutionized quantitative trading by enabling the discovery of complex patterns in financial data that traditional statistical methods cannot detect. This chapter covers the most effective ML architectures for trading applications.

Time Series Forecasting Models

🔷 LSTM (Long Short-Term Memory)

Best For: Sequential pattern recognition, price prediction, trend analysis

Architecture: Recurrent neural network with memory cells that can learn long-term dependencies

Key Advantages:

Handles variable-length sequences naturally
Learns long-term dependencies through memory gates
Resistant to vanishing gradient problem
Effective for multi-step ahead forecasting

Trading Applications:

Price movement prediction (next 1-hour, 1-day, etc.)
Volatility forecasting for risk management
Market regime detection and classification
Order book dynamics prediction

import torch
import torch.nn as nn

class LSTMPredictor(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(LSTMPredictor, self).__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers,
                           batch_first=True, dropout=0.2)
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        # x shape: (batch, seq_len, input_size)
        lstm_out, _ = self.lstm(x)
        # Take last output
        predictions = self.fc(lstm_out[:, -1, :])
        return predictions

# Example usage
model = LSTMPredictor(input_size=50, hidden_size=128,
                      num_layers=3, output_size=1)
# Input: 50 features, predict next price return

🔷 GRU (Gated Recurrent Unit)

Best For: Faster training than LSTM, similar performance with fewer parameters

Architecture: Simplified version of LSTM with fewer gates, computationally more efficient

Key Advantages:

Faster training and inference than LSTM
Fewer parameters, less prone to overfitting
Often matches LSTM performance in practice
Better for real-time trading with latency constraints

When to Use GRU vs LSTM:

Use GRU when computational efficiency is critical
Use LSTM when maximum accuracy is needed
Try both and compare on validation data

🔷 Transformer Models

Best For: Multi-asset prediction, incorporating diverse data sources, attention mechanisms

Architecture: Self-attention mechanism that processes entire sequences in parallel

Key Advantages:

Parallel processing for faster training
Attention mechanism identifies important time steps
Excellent for multi-modal data (prices, news, sentiment)
State-of-the-art results on many forecasting tasks

Trading Applications:

Multi-asset portfolio prediction
Combining price data with alternative data sources
Event-driven trading (earnings, news releases)
Cross-market correlation modeling

import torch.nn as nn

class TransformerPredictor(nn.Module):
    def __init__(self, d_model=64, nhead=8, num_layers=6):
        super().__init__()
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=d_model, nhead=nhead,
            dim_feedforward=256, dropout=0.1
        )
        self.transformer = nn.TransformerEncoder(
            encoder_layer, num_layers=num_layers
        )
        self.fc = nn.Linear(d_model, 1)

    def forward(self, x):
        # x: (seq_len, batch, features)
        transformer_out = self.transformer(x)
        prediction = self.fc(transformer_out[-1])
        return prediction

Ensemble Methods

Random Forest for Classification

Predict trading signal direction: Buy (+1), Hold (0), Sell (-1)

Robust to overfitting through bootstrapping
Feature importance analysis
Works well with hundreds of technical indicators
Handles non-linear relationships naturally

from sklearn.ensemble import RandomForestClassifier

# Train signal classifier
rf = RandomForestClassifier(
    n_estimators=500,
    max_depth=10,
    min_samples_split=100,
    max_features='sqrt',
    random_state=42
)

# Features: 100+ technical indicators
# Target: -1 (sell), 0 (hold), 1 (buy)
rf.fit(X_train, y_train)

# Feature importance
importances = rf.feature_importances_
top_features = sorted(zip(feature_names, importances),
                     key=lambda x: x[1], reverse=True)[:20]

XGBoost for Regression

Predict continuous returns for ranking and portfolio construction

State-of-the-art gradient boosting implementation
Fast training with GPU acceleration
Built-in regularization to prevent overfitting
Excellent for large datasets

import xgboost as xgb

# Predict next period return
model = xgb.XGBRegressor(
    n_estimators=1000,
    max_depth=6,
    learning_rate=0.01,
    subsample=0.8,
    colsample_bytree=0.8,
    gamma=1,
    tree_method='gpu_hist'
)

model.fit(X_train, y_train,
         eval_set=[(X_val, y_val)],
         early_stopping_rounds=50,
         verbose=False)

Reinforcement Learning

Deep Q-Network (DQN)

Learn optimal trading actions through trial and error

                RL Framework for Trading:
                State: Market features, positions, portfolio value
Actions: Buy, Sell, Hold (or continuous position sizing)
Reward: Profit/loss, Sharpe ratio, risk-adjusted returns
Policy: Learned strategy mapping states to actions

            

Advantages of RL in Trading:

Learns from interaction with market environment
Optimizes for long-term cumulative returns
Naturally handles sequential decision making
Can learn complex strategies without explicit rules
Adapts to changing market conditions through continuous learning

Popular RL Algorithms for Trading:

DQN: Deep Q-Networks for discrete actions
A3C: Asynchronous Advantage Actor-Critic
PPO: Proximal Policy Optimization (most stable)
SAC: Soft Actor-Critic for continuous actions
TD3: Twin Delayed DDPG for robust learning

Feature Engineering

Price-Based Features

Returns: log returns, percentage returns, standardized returns
Volatility: realized volatility, GARCH predictions
Moving averages: SMA, EMA, weighted MA
Price ratios: high/low, close/open

Technical Indicators

Momentum: RSI, MACD, Stochastic, Williams %R
Trend: ADX, Aroon, Parabolic SAR
Volume: OBV, Money Flow Index, Volume oscillator
Volatility: Bollinger Bands, ATR, Keltner Channels

Market Microstructure Features

Order book imbalance: bid-ask volume ratio
Trade flow: buyer-initiated vs seller-initiated trades
Spread: bid-ask spread, effective spread
Depth: cumulative volume at top N levels

Alternative Data Features

Sentiment scores from news and social media
Search trends and web traffic
Satellite imagery for retail, agriculture, energy
Credit card transactions for consumer spending

import pandas as pd
import ta  # Technical analysis library

def engineer_features(df):
    """Create comprehensive feature set"""

    # Price features
    df['returns'] = df['close'].pct_change()
    df['log_returns'] = np.log(df['close'] / df['close'].shift(1))
    df['volatility'] = df['returns'].rolling(20).std()

    # Technical indicators
    df['rsi'] = ta.momentum.rsi(df['close'], window=14)
    df['macd'] = ta.trend.macd_diff(df['close'])
    df['bb_high'] = ta.volatility.bollinger_hband(df['close'])
    df['bb_low'] = ta.volatility.bollinger_lband(df['close'])

    # Volume indicators
    df['obv'] = ta.volume.on_balance_volume(df['close'], df['volume'])
    df['mfi'] = ta.volume.money_flow_index(df['high'], df['low'],
                                           df['close'], df['volume'])

    # Lag features
    for lag in [1, 2, 5, 10, 20]:
        df[f'return_lag_{lag}'] = df['returns'].shift(lag)

    return df.dropna()

Model Training Best Practices

Time Series Cross-Validation

Never use random splits for time series! Use walk-forward validation:

Train on [T0, T1], validate on [T1, T2]
Train on [T0, T2], validate on [T2, T3]
Continues expanding window or rolling window

Preventing Overfitting

Regularization: L1, L2, dropout for neural networks
Early Stopping: Stop training when validation error increases
Ensemble Methods: Average multiple models to reduce variance
Feature Selection: Remove redundant and noisy features
Simpler Models: Start simple, increase complexity only if needed

Hyperparameter Optimization

Grid search for small parameter spaces
Random search for larger spaces
Bayesian optimization for expensive models
Use proper time series CV for evaluation

                Golden Rule: A simple model that works out-of-sample is infinitely better than a complex model that only works in-sample. Always prioritize generalization over training performance.
            