Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

Compare foundational deep learning optimizers like SGD and Adam to understand their impact on model training. This pack demonstrates how to implement and run both, highlighting their practical application in PyTorch for efficient model development.

beginner5 min4 steps

The play

Set Up Your Environment and Define a Simple Model
Begin by importing necessary PyTorch libraries. Define a basic neural network model, such as a simple linear regression, to serve as our training target.
Prepare Data and Loss Function
Generate synthetic input and target data. Instantiate your `SimpleModel` twice to compare optimizers independently. Define a suitable loss function, such as Mean Squared Error (MSE), for your task.
Initialize Different Optimizers
Create instances of `torch.optim.SGD` and `torch.optim.Adam`. Assign each to a different model instance, ensuring they operate on separate sets of model parameters. Experiment with different learning rates (`lr`).
Run Training Loops and Observe Loss
Implement a basic training loop for each optimizer. In each epoch, perform a forward pass, calculate loss, execute `loss.backward()` for gradient computation, and call `optimizer.step()` to update model parameters. Observe how the loss changes for each optimizer.

Starter code

import torch
import torch.nn as nn
import torch.optim as optim

# Define a simple model
class SimpleModel(nn.Module):
    def __init__(self):
        super(SimpleModel, self).__init__()
        self.linear = nn.Linear(10, 1)

    def forward(self, x):
        return self.linear(x)

# Dummy data
X = torch.randn(100, 10)
y = torch.randn(100, 1)

# Instantiate models
model_sgd = SimpleModel()
model_adam = SimpleModel()

# Define loss function
criterion = nn.MSELoss()

# SGD Optimizer
optimizer_sgd = optim.SGD(model_sgd.parameters(), lr=0.01)

# Adam Optimizer
optimizer_adam = optim.Adam(model_adam.parameters(), lr=0.001)

print("Training with SGD:")
for epoch in range(5):
    optimizer_sgd.zero_grad()
    outputs = model_sgd(X)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer_sgd.step()
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

print("\nTraining with Adam:")
for epoch in range(5):
    optimizer_adam.zero_grad()
    outputs = model_adam(X)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer_adam.step()
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

Source

Paperarxiv.org