datarekha
Deep Learning Medium

What regularization techniques do you know for deep networks, and how do they prevent overfitting?

For Data ScientistML Engineer
The short answer

Common techniques include L1 and L2 weight penalties, dropout (randomly zeroing activations), early stopping, data augmentation, and label smoothing. They reduce overfitting by constraining model capacity or adding noise so the network cannot memorize the training set.

How to think about it

Common techniques include L1 and L2 weight penalties, dropout (randomly zeroing activations), early stopping, data augmentation, and label smoothing. They reduce overfitting by constraining model capacity or adding noise so the network cannot memorize the training set.

Learn it properly Dropout, BN, LN

Keep practising

What causes overfitting in deep neural networks and how do you fight it? What is L2 regularisation (weight decay), and how does it reduce overfitting? What is early stopping, and how does it prevent overfitting? What are overfitting and underfitting, and how do you fix each? How do you train a deep learning model when you have very little labelled data?
All Deep Learning questions

Explore further

L1, L2, Elastic Net Overfitting & bias–variance Ridge Regression & Regularization
Regularization Dropout Overfitting L1 Regularization (Lasso) scikit-learn
Skip to content