Rectified Linear Unit

The rectified linear unit (ReLU) is a mathematical function that outputs a value equal to the input for positive inputs and outputs a 0 otherwise.

We can define ReLU using the following equation:

$$f(x)=\text{max}(0,x)$$

The derivative of the ReLU is 1 for positive inputs, 0 for negative inputs, and undefined for inputs of 0.

$${\displaystyle f^{\text{'}}(x)=\{\begin{array}{ll}1& \text{if }x>0\\ 0& \text{if }x<0\end{array}}$$

Implementing ReLU in numpy

import numpy as np

def relu(x):
    return np.maximum(0, x)

print(f"max(0,-20) = {relu(-20)}")

---

This page references the following sources:

• DeepLearning.AI Deep Learning Specialization (course 1, week 3)
• Google Machine Learning Glossary
• Wikipedia's "Rectifier (neural networks)" article
• numpy API reference (maximum)