What is a neural network and how does forward propagation work mathematically?

A neural network is a parameterised function composed of stacked layers. Each layer applies a linear transformation followed by a non-linear activation: h = σ(Wx + b), where W is a weight matrix, b is a bias vector, and σ is an activation function. Stacking L such layers gives a universal function approximator capable of learning arbitrarily complex input–output mappings, provided the network is wide or deep enough.

Forward propagation simply evaluates this composed function left to right: the input x passes through layer 1, the output becomes the input to layer 2, and so on until the final layer produces a prediction. The entire computation is a directed acyclic graph (DAG) of tensor operations — exactly the structure PyTorch's autograd engine records to enable automatic differentiation.

import torch
import torch.nn as nn

class TwoLayerNet(nn.Module):
    def __init__(self, in_dim, hidden_dim, out_dim):
        super().__init__()
        self.fc1 = nn.Linear(in_dim, hidden_dim)   # W1, b1
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, out_dim)  # W2, b2

    def forward(self, x):
        h = self.relu(self.fc1(x))  # h = ReLU(W1 x + b1)
        return self.fc2(h)           # y = W2 h + b2

model = TwoLayerNet(in_dim=10, hidden_dim=64, out_dim=1)
x = torch.randn(32, 10)   # batch of 32 inputs
y_hat = model(x)           # forward pass — calls model.forward(x)
print(y_hat.shape)         # torch.Size([32, 1])

Why depth matters: a network with one wide hidden layer can theoretically approximate any function (universal approximation theorem), but deeper networks can represent certain functions exponentially more efficiently — a function that needs an exponentially wide shallow network may be captured by a compact deep one, because each layer can reuse and compose features built by earlier layers.