Set your Publisher ID in src/config/ads.ts to display live ads.
Activation Functions, Loss & Backpropagation
ReLU, Sigmoid, Softmax, Cross-Entropy Loss, and Gradient Descent
Backpropagation uses the calculus chain rule to calculate the partial derivative of the loss function with respect to every weight and bias, guiding gradient descent updates.
Key Modern Activation Functions
ReLU (Rectified Linear Unit): f(x) = max(0, x). The gold standard for hidden layers because its derivative is 1 for positive inputs, eliminating the vanishing gradient problem in deep networks.
Sigmoid: f(x) = 1 / (1 + e^-x). Squashes values between 0 and 1, ideal for binary classification output layers.
Softmax: Normalizes a vector of raw logits into a valid probability distribution where all elements sum to 1.0, standard for multi-class classification.
# Learning rate eta controls step size along the negative gradient
learning_rate = 0.05
weight = 2.5
# Loss function: L(w) = (w - 1.0)^2
# Derivative: dL/dw = 2 * (w - 1.0)
for epoch in range(1, 6):
gradient = 2 * (weight - 1.0)
weight = weight - (learning_rate * gradient)
loss = (weight - 1.0) ** 2
print(f"Epoch {epoch}: Weight = {weight:.4f}, Loss = {loss:.6f}")The vanishing gradient problem occurs in deep networks using Sigmoid or Tanh, where repeated chain-rule multiplication of small derivatives (< 0.25) causes gradients in early layers to shrink to zero.
Set your Publisher ID in src/config/ads.ts to display live ads.
Test Yourself With Exercise
Which activation function is most widely used in hidden layers of modern deep networks to avoid the vanishing gradient problem?