Overfitting and Regularization in Machine Learning

## Part 1: Why Do We Need Regularization in Machine Learning? Machine learning models are remarkably good at minimizing training loss — that is precisely where the trouble begins. Give a flexible model enough capacity and enough time, and it will memorize the training set almost perfectly, including the noise, the mislabels, and every accidental pattern that happens to be there. The model then fails not because it learned too little, but because it learned too *much of the wrong thing* — training loss keeps dropping while test loss turns around and climbs. Regularization — the practice of deliberately constraining what the model is allowed to learn — is where we bridge the gap between memorizing the past and predicting the future. In this section we'll talk about how the unregularized baseline, chasing lower training loss, and regularization differ — and why constraining the model, counter-intuitively, is often what finally makes it work outside the training set. ### The Unregularized Baseline: Intuition The simplest approach trains a flexible model on the full training set and measures progress by how much training loss falls. The optimizer follows the gradient to the lowest point it can find, and the dashboard celebrates every decimal: > Train → train loss drops → celebrate → deploy → test loss is much worse → ? This works when the data is clean, abundant, and the model's capacity is small. But it assumes: - The training set is a perfect reflection of the real distribution the model will face. - Every pattern the model can represent is a pattern worth representing. - The lowest training loss corresponds to the lowest generalization error. For a tiny polynomial fit with lots of data, these assumptions often hold. For a neural network with millions of parameters on noisy labels, they never do. ### Chasing Training Loss: Guiding with More Capacity An obvious fix is to make the model bigger. If it hasn't learned the pattern yet, why not add more layers, more features, more epochs? > More parameters → lower training loss → better generalization? This approach helps for a while, but it breaks down quickly: - Training loss can always be driven lower, given enough parameters — even to zero, by memorization [1]. - Generalization error stops falling well before training loss bottoms out, then turns upward as the model begins fitting noise [2]. - Adding capacity is silent about *what* the model is learning — it only measures how well it fits, not what it is memorizing. More capacity is a tool for expressiveness, not for generalization. The question becomes: why does the model know the training set by heart but fail on a single new example? ### Regularization: Constrain First, Then Learn Regularization answers that question with a two-stage contract. Instead of minimizing training loss alone, you minimize a combined objective: 1. **Empirical loss**: the standard training error that pushes the model to fit the data. 2. **Penalty**: an explicit cost on complexity — large weights, many features, long training — that pushes the model toward simpler, more stable solutions. This approach: - Keeps the effective capacity in line with the data available, so generalization error tracks training error for longer. - Turns the capacity-vs-data balance from an accident of architecture into an explicit hyperparameter you can tune. - Preserves the model's flexibility at inference time — it still has millions of parameters, it's just not free to use all of them wildly. ### The Common Feeling: "Isn't This Just Making the Model Worse?" Many practitioners (myself included, at first) find regularization counter-intuitive. If you squint, it looks like we just: - Added a term to the loss that has nothing to do with the data. - Forced the optimizer to stop at a higher training loss than it could otherwise reach. Isn't that deliberately choosing a worse fit to the training data? In practice, yes — the training loss *does* get worse. But conceptually, there are two crucial differences: 1. **What you're optimizing:** - Unregularized: you optimize for the past — the training set alone. - Regularized: you optimize for the future — the training set plus a preference for stable, general solutions. 2. **Failure mode:** - Unregularized failure is silent — training loss keeps falling, test loss climbs, and the gap hides in plain sight. - Regularized failure is visible — the training-test gap shrinks, and underfitting, when it happens, is obvious from both sides. It's a subtle but important shift: from asking "how low can we get training loss?" to asking "how much of this low training loss is actually useful?" ### Why the Distinction Matters in Real Applications This difference is especially sharp for the workloads where flexibility is tempting: small labeled datasets, noisy labels, or high-dimensional features where memorization is cheap and generalization is rare. - **Unregularized setup:** A model for customer churn trained without regularization achieves 98% training accuracy but 62% test accuracy — the gap is the model reciting training-set peculiarities. - **Regularized setup:** The same architecture with L2 weight decay and early stopping drops training accuracy to 86% but raises test accuracy to 82% — the gap narrows, and the deployed model actually works. Formally, generalization error decomposes as [2]: > E[(f_θ(x) − y)²] = bias² + variance + σ² Where *bias* measures how far the model class is from the true function, *variance* measures how much predictions move when you retrain on a new sample, and *σ²* is irreducible noise. More capacity lowers bias but raises variance; regularization controls the variance side of the trade. This is why regularization is the bridge between research and deployment — without it, every model is a good student who memorized the practice exam. ### Advanced Variants: Beyond L2 When plain weight decay isn't enough, richer forms of regularization address the specific way the model is overfitting: - **Early stopping**: treat the number of training steps as a capacity knob — stop when validation loss stops falling [4]. - **Dropout**: randomly zero out activations during training so the model cannot lean on any single neuron [5]. - **Data augmentation**: multiply the effective training set by applying transformations that preserve the label — the strongest regularizer for images and audio. - **L1 and sparsity**: push weights toward zero entirely, so the model selects a smaller feature set rather than using everything weakly [3]. The guidance is structural: different forms of regularization target different failure modes — diagnose which kind of overfitting you have before choosing which tool. ### Intuition - Unregularized: "Fit the training set as well as you can." - More capacity: "Fit the training set as well as you can, with a bigger toolbox." - Regularized: "Fit the training set as well as you can — but only with solutions that look simple." ### Conclusion Regularization in machine learning can feel, at first, like an unnecessary punishment — why force the model to accept a worse fit to the data you have? But the shift in what you optimize and the explicit control over the training-test gap are what make it different from simply using a smaller model. For abundant, clean data, unregularized training is sufficient. For noisy or limited data, regularization unlocks generalization the data alone cannot provide. For modern deep models, dropout and early stopping show the full potential: enormous models that somehow still generalize, because capacity and simplicity are no longer at odds. That's why regularization — in one form or another — remains central to shaping how models not only fit the data, but also carry what they learned into the future. ## Part 2: How to Train Models That Generalize This is a recap of my working notes from tuning models on limited and noisy datasets, cross-checked against The Elements of Statistical Learning [3] and the modern discussion opened by Zhang et al.'s memorization experiments [1]. If you find this interesting, I highly recommend going through both. ### Generalization Setup in the ML Context - Training set (D_train): the data the optimizer sees - Validation set (D_val): held-out data used to choose hyperparameters and stopping points - Test set (D_test): touched once, at the end, to estimate real performance - Model class (H): the set of functions the architecture can express - Regularization strength (λ): the knob trading training fit against solution simplicity - Generalization gap: test loss minus training loss — the number that tells the truth Unlike changing the architecture, regularization modifies the objective and the training protocol — the same network can go from memorizing to generalizing with a schedule and a penalty term. ### Naive Training: Minimize the Training Loss The simplest protocol trains until the loss stops improving on the training set itself. The core objective is: > min_θ L_train(θ) = (1/N) Σ loss(f_θ(x_i), y_i) Interpretation: find the parameters that best explain the data you have, and hope they transfer to data you don't. - If the model class is small relative to the data (e.g., linear regression on millions of rows), this works. - If the model class is large relative to the data (deep nets, wide feature sets, small samples), this reliably fails. This is like grading students on the exact questions they practiced — perfect scores, no way to tell who understood anything. The problems: - **Noise memorization**: with enough capacity the model fits mislabeled and noisy examples exactly, encoding errors as if they were signal [1]. - **No stopping signal**: training loss falls monotonically, so it can never tell you when to stop — it always says "keep going." - **Silent leakage**: without a disciplined split, information from the test set seeps into choices about features and hyperparameters, inflating every estimate. As I noted in my experiment logs: a model that is never evaluated on data it hasn't seen is not being evaluated at all — it's being rehearsed. ### Regularized Training: Penalty, Patience, and Protocol To train models that transfer, we intervene on the objective, the schedule, and the data — three independent levers: - **Objective level — weight penalties**: add λ·‖θ‖² (ridge/L2) for stability or λ·‖θ‖₁ (lasso/L1) for sparsity, shrinking the solution toward simplicity [3]. - **Schedule level — early stopping**: monitor validation loss each epoch and stop when it has not improved for a set patience — capacity control measured in steps rather than parameters [4]. - **Data level — augmentation and dropout**: enlarge the effective dataset with label-preserving transforms, and prevent co-adaptation by dropping activations at random during training [5]. - **Protocol level — honest splits**: choose λ and every other hyperparameter on validation folds; touch the test set exactly once. Intuition: don't just train harder; decide what counts as a simple solution, let the validation set arbitrate, and keep the test set out of the room while you argue. ### The Generalization Workflow Algorithm Algorithm: Regularized Training Workflow 1. Split the data — train, validation, test — before any exploration; freeze the test set 2. Train an unregularized baseline and record the train-validation gap: this is the size of your overfitting problem 3. Choose the regularizer to match the failure: L2 for instability, L1 for feature bloat, augmentation for small data, dropout for co-adapting deep nets 4. Sweep λ on validation folds — from underfitting (both losses high) to overfitting (gap wide) — and plot the curve 5. Add early stopping with patience; let the validation set, not the epoch budget, decide when training ends 6. Retrain on train+validation with the chosen settings, then evaluate on the untouched test set — once 7. Monitor in production: data drifts, and a model regularized for yesterday's distribution can quietly start underfitting today's #### Key insights: - Step 2: The gap, not the training loss, is the diagnostic — a large gap says overfit, twin high losses say underfit, and the remedies point in opposite directions. - Step 4: The validation curve is U-shaped — the goal is the bottom of the U, and folklore values of λ are someone else's bottom. - Step 6: A test set consulted twice is a validation set — its estimate is spent the moment it influences a decision. ### The Role of Capacity and Validation Discipline In generalization work, model capacity and validation discipline each serve distinct roles in keeping performance estimates honest: **Model Capacity** - Defined by what the architecture can express: parameters, depth, feature count, training steps. - Used to ensure the model *can* represent the true pattern — too little capacity and no amount of data helps. - Intuition: "Make the model powerful enough to be right, then constrain it enough to stop being clever." **Validation Discipline** - Typically a three-way split, with cross-validation when data is scarce; the test set is sacred and single-use. - Why? Because every decision made while looking at a dataset leaks information into the model — discipline is what keeps the final number believable. - Intuition: "The validation set is your advisor; the test set is your exam. Never let the advisor take the exam." **Why Both Matter** - The right capacity makes good solutions reachable; the right discipline tells you truthfully whether you reached one. - A perfectly tuned model behind a leaky protocol reports numbers that evaporate in production. - Too little capacity underfits no matter the protocol; too little discipline overfits the *evaluation itself* — the most expensive kind of overfitting, because it is discovered by your users. ### Analogy to Collecting More Data Collecting more data also fights overfitting, but with key differences: - **More data**: reduces variance by giving the model more of reality to average over — permanent, but slow and often expensive. - **Regularization**: reduces variance by restricting the model's freedom — immediate and free, but bounded by how much structure the penalty correctly assumes. In practice they are complements, not competitors: regularize to survive the data you have, collect to deserve the model you want — and revisit λ every time the dataset grows. ### TL;DR: - Capacity: Powerful enough to represent the truth, constrained enough not to memorize the noise → tune the penalty, not just the architecture. - Validation Discipline: Every peek at held-out data spends some of its honesty → three-way splits, single-use test sets, decisions on validation only. ### A Toy Example: Polynomial Curve Fitting Let's walk through the classic toy environment: 30 noisy samples from a smooth curve, a degree-15 polynomial with far too much capacity, and ridge regularization to keep it honest. #### Model Design ```python import numpy as np def fit_ridge(X, y, lam=1.0): n_features = X.shape[1] A = X.T @ X + lam * np.eye(n_features) return np.linalg.solve(A, X.T @ y) # closed-form ridge solution ``` #### Key Protocol Functions Sweep the regularization strength on the validation set (simplified): ```python def sweep_lambda(X_tr, y_tr, X_val, y_val, lams): best_lam, best_err = None, float("inf") for lam in lams: # e.g., np.logspace(-6, 2, 25) w = fit_ridge(X_tr, y_tr, lam) err = np.mean((X_val @ w - y_val) ** 2) if err < best_err: best_lam, best_err = lam, err return best_lam, best_err ``` Measure what actually matters — the gap (simplified): ```python def evaluate_generalization(w, X_tr, y_tr, X_te, y_te): train_mse = float(np.mean((X_tr @ w - y_tr) ** 2)) test_mse = float(np.mean((X_te @ w - y_te) ** 2)) gap = test_mse - train_mse # the honest number return {"train_mse": train_mse, "test_mse": test_mse, "gap": gap} ``` At λ = 0 the degree-15 polynomial threads every noisy point — train MSE near zero, test MSE enormous. At the swept optimum, train MSE rises slightly and test MSE collapses: the model gave up memorizing and started generalizing. ### Conclusions - Gap vs performance: The train-test gap, not the training loss, is the health metric — a regularized model with a small gap beats a memorizer with a perfect fit. - Validation grounding: Choosing λ, stopping points, and features on validation data turns generalization from a hope into a measured, tunable property. - Flexibility: Penalties, patience, and dropout rates are configuration — far cheaper to iterate than architectures, and often worth more. - Scaling: The toy code fits polynomials, but production generalization work involves cross-validated sweeps, drift-triggered retuning, augmentation pipelines, and the discipline to leave the test set alone. More capacity can only make memorization easier. Regularization decides what kind of solutions deserve that capacity, and validation discipline tells you honestly whether it worked. While toy demos like "polynomial curve fitting" are simple, the mechanics mirror how regularized training keeps real models useful on the only data that matters — the data they have never seen. ## References 1. Zhang, C., et al. "Understanding Deep Learning Requires Rethinking Generalization." ICLR (2017). 2. Geman, S., Bienenstock, E., and Doursat, R. "Neural Networks and the Bias/Variance Dilemma." Neural Computation (1992). 3. Hastie, T., Tibshirani, R., and Friedman, J. "The Elements of Statistical Learning." Springer (2009). 4. Prechelt, L. "Early Stopping — But When?" Neural Networks: Tricks of the Trade, Springer (1998). 5. Srivastava, N., et al. "Dropout: A Simple Way to Prevent Neural Networks from Overfitting." JMLR (2014). 6. Hoerl, A., and Kennard, R. "Ridge Regression: Biased Estimation for Nonorthogonal Problems." Technometrics (1970). 7. Tibshirani, R. "Regression Shrinkage and Selection via the Lasso." JRSS-B (1996). 8. Belkin, M., et al. "Reconciling Modern Machine-Learning Practice and the Classical Bias-Variance Trade-off." PNAS (2019).