Activation Functions Explained: Sigmoid, Tanh, ReLU, and Leaky ReLU (and When to Actually Use Them)

2026-07-19 · Source: Deep Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Novice, medium

Summary

The article explains four common activation functions: Sigmoid, Tanh, ReLU, and Leaky ReLU, detailing their formulas, behavior, advantages, disadvantages, and typical use cases. It emphasizes that activation functions introduce non-linearity, preventing deep networks from collapsing into a single linear layer. Sigmoid, which squashes inputs to 0-1, is ideal for binary classification output layers but suffers from vanishing gradients and non-zero-centered output in hidden layers. Tanh, a zero-centered version mapping to -1 to 1, improves convergence but still faces vanishing gradients. ReLU, defined as max(0, x), became the default for hidden layers due to its computational efficiency and lack of vanishing gradients on the positive side, though it can suffer from "dying ReLU" neurons. Leaky ReLU addresses this by allowing a small alpha * x for negative inputs, preventing neurons from dying, and is often used when dying ReLU issues arise, particularly in GANs. The article provides tf.keras examples for each.

Key takeaway

For Machine Learning Engineers designing neural networks, your choice of activation function is critical for training stability and model performance. Start with ReLU for hidden layers, but be prepared to switch to Leaky ReLU if you encounter dying neurons. Always use Sigmoid for binary classification output layers, and consider Tanh/Sigmoid combinations for recurrent network gates to ensure efficient learning and prevent issues like vanishing gradients.

Key insights

Activation functions introduce non-linearity, enabling neural networks to learn complex patterns beyond linear regression.

Principles

• Non-linearity is crucial for deep network learning.
• Vanishing gradients impede deep network training.
• Zero-centered outputs stabilize gradient flow.

Method

Select activation functions based on layer type and problem: Sigmoid for binary classification output, ReLU for hidden layers, Leaky ReLU to mitigate dying neurons, and Tanh/Sigmoid for RNN gates.

In practice

• Use Sigmoid for binary classification output layers.
• Default to ReLU for most hidden layers.
• Employ Leaky ReLU if dying neurons occur.

Topics

• Activation Functions
• Neural Networks
• ReLU
• Vanishing Gradients
• Binary Classification
• TensorFlow

Code references

• codebasics/deep-learning-keras-tf-tutorial

Best for: AI Student, Machine Learning Engineer, Data Scientist

Related on AIssential

• tanh vs relu #datascience #machinelearning #mathematics #statistics — Tanh suffers from vanishing gradients, while ReLU avoids this but can lead to dying neurons.
• Activation Functions in Neural Networks - Explained — Activation functions introduce non-linearity, enabling neural networks to learn complex decision boundaries.
• Activation Functions: The Hidden Switch Behind Every Neural Network — Activation functions are crucial for neural networks to learn complex, non-linear patterns, transforming simple linear operations into capabilities for real-world tasks.
• Making Neural Networks Learn Better: Understanding Activation Functions, Xavier Initialization, He… — Effective deep neural network training relies on proper activation functions, weight initialization, and Batch Normalization.
• The 60-Year Hunt for AI's Most Important Function — Activation functions are critical for neural network expressiveness, evolving from simple thresholds to complex gated, smooth functions.
• Separation Power of Equivariant Neural Networks — Non-polynomial activations in equivariant networks offer equivalent, maximal separation power, simplifying architectural choices.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Deep Learning on Medium.