What is the mathematical setup of a Generative Adversarial Network (GAN) and what training challenges do they have?

A GAN consists of two competing networks: a generator G that maps random noise z ~ p(z) to fake data samples, and a discriminator D that classifies inputs as real or fake. They play a minimax game with objective: min_G max_D E[log D(x)] + E[log(1 - D(G(z)))]. At the Nash equilibrium, G produces samples from the true data distribution and D outputs 0.5 for every input (cannot distinguish real from fake).

In practice, GANs suffer from several well-known training challenges: mode collapse (G learns to produce only a subset of modes of the data distribution); training instability (the minimax game does not converge reliably); and vanishing generator gradient (when D becomes too good early on, it correctly classifies fake samples with near-certainty, giving G near-zero gradient signal). These led to many GAN variants — DCGAN (convolutional architecture), WGAN (Wasserstein distance instead of JS divergence), and progressive growing GANs.

import torch
import torch.nn as nn

latent_dim, img_dim = 100, 784

# Generator: noise -> fake image
generator = nn.Sequential(
    nn.Linear(latent_dim, 256), nn.ReLU(),
    nn.Linear(256, 512), nn.ReLU(),
    nn.Linear(512, img_dim), nn.Tanh()
)

# Discriminator: image -> real (1) or fake (0)
discriminator = nn.Sequential(
    nn.Linear(img_dim, 512), nn.LeakyReLU(0.2),
    nn.Linear(512, 256), nn.LeakyReLU(0.2),
    nn.Linear(256, 1)  # raw logit; use BCEWithLogitsLoss
)

criterion = nn.BCEWithLogitsLoss()
opt_G = torch.optim.Adam(generator.parameters(),     lr=2e-4, betas=(0.5, 0.999))
opt_D = torch.optim.Adam(discriminator.parameters(), lr=2e-4, betas=(0.5, 0.999))

for real_imgs, _ in loader:
    real_imgs = real_imgs.view(-1, img_dim)
    bs = real_imgs.size(0)

    # Train Discriminator
    z = torch.randn(bs, latent_dim)
    fake_imgs = generator(z).detach()  # detach: don't update G here
    loss_D = (criterion(discriminator(real_imgs), torch.ones(bs, 1))
            + criterion(discriminator(fake_imgs), torch.zeros(bs, 1)))
    opt_D.zero_grad(); loss_D.backward(); opt_D.step()

    # Train Generator
    z = torch.randn(bs, latent_dim)
    loss_G = criterion(discriminator(generator(z)), torch.ones(bs, 1))
    opt_G.zero_grad(); loss_G.backward(); opt_G.step()