Chapter 1: Introduction to Reinforcement Learning

1.1 What is Reinforcement Learning?

Reinforcement Learning (RL) is a branch of machine learning where an agent learns to make decisions by interacting with an environment. Unlike supervised learning, which relies on labeled examples, or unsupervised learning, which finds patterns in unlabeled data, reinforcement learning learns through trial and error, receiving feedback in the form of rewards or penalties.

The reinforcement learning framework consists of three fundamental components:

• Agent: The learner or decision maker
• Environment: The world the agent interacts with
• Reward: The feedback signal that guides learning

The Agent-Environment Interface

At each time step t, the agent:

1. Observes the current state sₜ of the environment
2. Selects and executes an action aₜ
3. Receives a reward rₜ₊₁
4. Observes the new state sₜ₊₁

┌─────────┐
│  Agent  │
└────┬────┘
     │ aₜ (action)
     ▼
┌─────────────┐
│ Environment │
└────┬────────┘
     │ sₜ₊₁, rₜ₊₁ (state, reward)
     ▼
┌─────────┐
│  Agent  │
└─────────┘

1.2 The Reinforcement Learning Problem

The goal of reinforcement learning is to find an optimal policy π* that maximizes the expected cumulative reward over time. This is formalized as maximizing the expected return:

Gₜ = Rₜ₊₁ + γRₜ₊₂ + γ²Rₜ₊₃ + ... = Σ γᵏRₜ₊ₖ₊₁

where:
- Gₜ is the return from time step t
- γ is the discount factor (0 ≤ γ ≤ 1)
- Rₜ is the reward at time t

The Discount Factor γ

The discount factor γ determines the present value of future rewards:

• γ = 0: Myopic agent, only considers immediate rewards
• γ → 1: Far-sighted agent, values future rewards almost as much as immediate ones
• γ = 0.9: Common choice, balances immediate and future rewards

1.3 Types of RL Problems

Episodic vs. Continuing Tasks

Model-Based vs. Model-Free RL

Model-Based RL: The agent has access to (or learns) a model of the environment's dynamics p(s', r | s, a). This allows planning by simulating future trajectories.

// Model-Based: Plan ahead using the model
function plan(state, model, depth):
    best_action = null
    best_value = -infinity

    for action in possible_actions:
        value = 0
        for next_state, reward, prob in model.predict(state, action):
            value += prob * (reward + γ * V(next_state))

        if value > best_value:
            best_value = value
            best_action = action

    return best_action

Model-Free RL: The agent learns directly from experience without building an explicit model. This is simpler but may require more samples.

// Model-Free: Learn Q-values directly from experience
function q_learning_update(s, a, r, s_next):
    Q[s][a] = Q[s][a] + α * (r + γ * max(Q[s_next]) - Q[s][a])

1.4 Exploration vs. Exploitation

One of the most fundamental dilemmas in RL is the exploration-exploitation tradeoff:

• Exploitation: Choose actions that have worked well in the past to maximize immediate reward
• Exploration: Try new actions to potentially discover better strategies

Epsilon-Greedy Strategy

A simple but effective approach to balance exploration and exploitation:

function epsilon_greedy(Q, state, epsilon):
    if random() < epsilon:
        // Explore: choose random action
        return random_action()
    else:
        // Exploit: choose best known action
        return argmax(Q[state])

Other Exploration Strategies

1.5 RL vs. Other ML Paradigms

Comparison with Supervised Learning

When to Use RL?

Reinforcement learning is particularly suitable when:

• There's no supervisor, only a reward signal
• Feedback is delayed, not instantaneous
• Time matters (sequential, non-i.i.d. data)
• Agent's actions affect subsequent data it receives
• The optimal action isn't known a priori

1.6 Real-World Applications

Game Playing

RL has achieved superhuman performance in games:

• Backgammon (TD-Gammon, 1992): First major RL success using temporal difference learning
• Atari Games (DQN, 2013): Deep Q-Networks learned to play from pixels
• Go (AlphaGo, 2016): Defeated world champion using deep RL and Monte Carlo Tree Search
• StarCraft II (AlphaStar, 2019): Mastered complex multi-agent strategy game

Robotics

// Robot Control with RL
class RobotAgent:
    def __init__(self):
        self.policy_network = PolicyNetwork()
        self.value_network = ValueNetwork()

    def act(self, observation):
        # Joint angles, velocities, tactile feedback
        state = self.process_sensors(observation)

        # Output: motor commands
        action = self.policy_network(state)
        return action

    def train(self, trajectory):
        # Learn from interaction with physical environment
        # Reward: task completion, energy efficiency, safety
        self.update_policy(trajectory)

Applications include:

• Manipulation and grasping
• Locomotion (walking, running)
• Navigation and path planning
• Human-robot interaction

Autonomous Vehicles

Self-driving cars use RL for:

• Lane keeping and following
• Merging and overtaking decisions
• Parking
• Interaction with other vehicles and pedestrians

Resource Management

• Data Centers: Google reduced cooling costs by 40% using RL
• Energy Grids: Optimize power distribution and renewable integration
• Traffic Control: Adaptive traffic light systems to reduce congestion
• Network Routing: Packet routing and load balancing

Finance

// Trading Agent
class TradingAgent:
    def __init__(self):
        self.portfolio = Portfolio()
        self.q_function = QNetwork()

    def observe(self, market_state):
        # State: prices, volumes, technical indicators, news sentiment
        return {
            'prices': market_state.prices,
            'portfolio': self.portfolio.holdings,
            'indicators': self.compute_indicators(market_state)
        }

    def trade(self, state):
        # Actions: buy, sell, hold for each asset
        actions = self.q_function.get_best_actions(state)

        # Reward: PnL, Sharpe ratio, risk-adjusted returns
        return actions

Healthcare

• Treatment Planning: Personalized treatment strategies
• Drug Dosing: Optimal medication schedules
• Clinical Trials: Adaptive trial designs
• Resource Allocation: Hospital bed and staff management

1.7 Challenges in Reinforcement Learning

Sample Efficiency

RL often requires millions of interactions to learn effective policies. This is problematic when:

• Real-world interactions are expensive (robotics)
• Safety is critical (autonomous vehicles, healthcare)
• Environment interaction is slow (physical systems)

Credit Assignment Problem

When a reward is received, which past actions were responsible? This is challenging when:

• Rewards are sparse (only at episode end)
• Delays are long (action → reward lag)
• Multiple agents are involved

Exploration in Large State Spaces

How can an agent efficiently explore when:

• State space is enormous or continuous
• Rewards are extremely sparse
• Many actions are available

Stability and Convergence

Deep RL can be unstable because:

• Non-stationary targets (moving goal posts)
• Correlated samples (sequential data)
• Bootstrapping (learning from estimates)
• Function approximation errors

1.8 The RL Toolbox

Popular RL Libraries

// Stable-Baselines3 (PyTorch)
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env

# Create environment
env = make_vec_env('CartPole-v1', n_envs=4)

# Initialize agent
model = PPO('MlpPolicy', env, verbose=1)

# Train
model.learn(total_timesteps=100000)

# Evaluate
obs = env.reset()
for _ in range(1000):
    action, _ = model.predict(obs)
    obs, reward, done, info = env.step(action)

Environment Standards

// OpenAI Gym Interface
import gym

class CustomEnv(gym.Env):
    def __init__(self):
        self.action_space = gym.spaces.Discrete(4)
        self.observation_space = gym.spaces.Box(
            low=0, high=255, shape=(84, 84, 3), dtype=np.uint8
        )

    def reset(self):
        # Reset environment to initial state
        return observation

    def step(self, action):
        # Execute action
        # Return (observation, reward, done, info)
        return obs, reward, done, info

    def render(self, mode='human'):
        # Visualize environment
        pass

Key Libraries and Frameworks

1.9 Getting Started with RL

Your First RL Agent

import numpy as np
import gym

# Create environment
env = gym.make('FrozenLake-v1')

# Initialize Q-table
Q = np.zeros([env.observation_space.n, env.action_space.n])

# Hyperparameters
alpha = 0.1      # Learning rate
gamma = 0.99     # Discount factor
epsilon = 0.1    # Exploration rate
episodes = 10000

# Training loop
for episode in range(episodes):
    state = env.reset()
    done = False

    while not done:
        # Epsilon-greedy action selection
        if np.random.random() < epsilon:
            action = env.action_space.sample()  # Explore
        else:
            action = np.argmax(Q[state])        # Exploit

        # Take action
        next_state, reward, done, _ = env.step(action)

        # Q-learning update
        Q[state, action] = Q[state, action] + alpha * (
            reward + gamma * np.max(Q[next_state]) - Q[state, action]
        )

        state = next_state

print("Training completed!")
print("Learned Q-table:")
print(Q)

1.10 The Road Ahead

This chapter introduced the fundamentals of reinforcement learning. In the following chapters, we'll dive deeper into:

• Chapter 2: Markov Decision Processes - The mathematical foundation
• Chapter 3: Value-Based Methods - Q-Learning and Deep Q-Networks
• Chapter 4: Policy Gradient - Direct policy optimization
• Chapter 5: Actor-Critic - Combining value and policy learning
• Chapter 6: Model-Based RL - Learning and planning with models
• Chapter 7: Multi-Agent RL - Coordinating multiple agents
• Chapter 8: Production Deployment - Scaling RL to real applications