Hyperparameter tuning

You’ve trained a model. Now you have to pick max_depth=5 or =7, learning_rate=0.01 or 0.05, n_estimators=200 or 500. The default “I’ll try a few by hand” approach is fine for exploration; it falls apart when you have five hyperparameters and a deadline.

This lesson is about doing it systematically without falling into the classic trap: tuning on the test set.

The three strategies

A surprising result from Bergstra & Bengio (2012): random search beats grid search for the same number of trials, because most hyperparameters don’t matter equally — grid wastes trials on the unimportant axes. So grid is mostly dead; the choice is random vs Bayesian. Random is simpler and works well; Bayesian (Optuna) is the right default once you’ve used it a few times.

Bergstra and Bengio (2012): for the same budget, random search covers the axis that matters far better than a grid that keeps re-testing the same few values.

Rule zero: tune inside a Pipeline, with cross-validation

You never tune on the test set. You also never tune on a single validation set if you can help it — cross-validation gives a more stable estimate. And every preprocessing step must live inside the Pipeline so each CV fold preprocesses cleanly (see the data-leakage lesson).

The pattern looks like:

pipe = Pipeline([("scaler", StandardScaler()), ("clf", model)])
search = GridSearchCV(pipe, param_grid, cv=5, scoring="roc_auc")
search.fit(X_train, y_train)
# Best params evaluated on X_test ONLY ONCE, after tuning is fully done
final_score = search.score(X_test, y_test)

The test set is used exactly once, at the very end. Every “should I try a different hyperparam?” decision happens against cross-validated training data only.

Grid search and random search in sklearn

import numpy as np
from scipy.stats import randint
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV

X, y = make_classification(n_samples=2000, n_features=15, n_informative=8, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", RandomForestClassifier(n_jobs=-1, random_state=0)),
])

# --- Grid search: every combination ---
grid = {
    "clf__n_estimators": [100, 300],
    "clf__max_depth":    [5, 10, None],
    "clf__max_features": ["sqrt", "log2"],
}
gs = GridSearchCV(pipe, grid, cv=5, scoring="roc_auc", n_jobs=-1).fit(X_tr, y_tr)
print(f"Grid   - {len(gs.cv_results_['params'])} fits | best CV AUC: {gs.best_score_:.3f}")
print("  best:", gs.best_params_)

# --- Random search: sample from distributions ---
dist = {
    "clf__n_estimators": randint(50, 500),
    "clf__max_depth":    randint(3, 15),
    "clf__max_features": ["sqrt", "log2", 0.5, 0.8],
}
rs = RandomizedSearchCV(pipe, dist, n_iter=20, cv=5, scoring="roc_auc",
                        n_jobs=-1, random_state=0).fit(X_tr, y_tr)
print(f"\nRandom - {rs.n_iter} fits | best CV AUC: {rs.best_score_:.3f}")
print("  best:", rs.best_params_)

# Test-set evaluation — exactly ONCE, with the chosen model
print(f"\nHeld-out test AUC: {rs.score(X_te, y_te):.3f}")

A few things worth noting in that snippet:

• pipe__clf__n_estimators syntax — step_name__param lets you tune parameters of any pipeline step.
• cv=5 does 5-fold CV inside the search.
• The held-out test set is touched once, at the end.

Bayesian tuning with Optuna

Optuna is the de facto modern default. You define an objective(trial) function that returns a score; Optuna samples hyperparameters intelligently and improves over trials. Unlike grid/random, it learns from previous results.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, cross_val_score

X, y = make_classification(n_samples=2000, n_features=15, n_informative=8, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)

# Real Optuna usage looks like:
#   import optuna
#   def objective(trial):
#       n_estimators = trial.suggest_int("n_estimators", 50, 500)
#       max_depth    = trial.suggest_int("max_depth", 3, 15)
#       max_features = trial.suggest_categorical("max_features", ["sqrt", "log2"])
#       model = RandomForestClassifier(n_estimators=n_estimators,
#                                      max_depth=max_depth, max_features=max_features,
#                                      n_jobs=-1, random_state=0)
#       return cross_val_score(model, X_tr, y_tr, cv=5, scoring="roc_auc").mean()
#   study = optuna.create_study(direction="maximize")
#   study.optimize(objective, n_trials=30)

# Stand-in: a minimal trial loop (random search) with the same structure
rng = np.random.default_rng(0)
trials = []
for t in range(15):
    params = {
        "n_estimators": int(rng.integers(50, 500)),
        "max_depth":    int(rng.integers(3, 15)),
        "max_features": str(rng.choice(["sqrt", "log2"])),
    }
    model = RandomForestClassifier(n_jobs=-1, random_state=0, **params)
    score = cross_val_score(model, X_tr, y_tr, cv=5, scoring="roc_auc").mean()
    trials.append((score, params))
    best_so_far = max(trials, key=lambda x: x[0])
    print(f"trial {t:2d}: {score:.4f}  best so far: {best_so_far[0]:.4f}")

best_score, best_params = max(trials, key=lambda x: x[0])
print(f"\nbest params: {best_params}")
print(f"best CV AUC: {best_score:.4f}")

The key insight: a real Optuna study would prune unpromising trials early (with MedianPruner), reuse evaluations across similar parameter regions (TPE), and let you express conditional spaces (e.g., only sample min_samples_split when max_depth > 5). For ten lines more code than random search you get materially better results, especially as the number of hyperparameters grows.

Common tuning ranges to start from

When you don’t know the right range, these are sensible defaults to search over:

Use log-uniform sampling for any parameter that spans orders of magnitude (learning_rate, C, regularization weights). Linear sampling wastes most of its trials on the high end.

Nested CV — for an honest estimate of generalization

If you tune hyperparameters using CV on your training data and then report the same CV score, that score is optimistic — the hyperparameters were chosen because they happen to work well on those folds. The real generalization estimate is lower.

The fix is nested cross-validation:

• Outer loop (e.g., 5 folds): produces unbiased generalization estimates.
• Inner loop (e.g., 5 folds within each outer training fold): does the hyperparameter search.

You end up with five outer scores from five separately-tuned models. Their mean is an honest estimate of how a tuned-on-fresh-data model would do.

from sklearn.model_selection import cross_val_score, GridSearchCV, KFold
outer = KFold(n_splits=5, shuffle=True, random_state=0)
inner = KFold(n_splits=5, shuffle=True, random_state=1)
search = GridSearchCV(pipe, param_grid, cv=inner, scoring="roc_auc")
nested_scores = cross_val_score(search, X, y, cv=outer, scoring="roc_auc")
print("honest CV AUC:", nested_scores.mean())

Nested CV is expensive (you train outer × inner × |grid| models). Use it when reporting numbers in a paper, a benchmark, or a critical business decision. For day-to-day iteration, the simpler “tune with CV on train, evaluate once on test” pattern is fine.

A practical recipe

For a new tabular problem, this is what to do:

1. Train a default RandomForest. Get a baseline CV score.
2. If you need more accuracy, switch to HistGradientBoosting / LightGBM with default params + early stopping. Get a new baseline.
3. Wrap in a Pipeline. Run RandomizedSearchCV (or Optuna) with 30-100 trials over the 4 key hyperparameters.
4. Evaluate the best pipeline on the held-out test set once.
5. If the gap between CV score and test score is large, suspect leakage or distribution shift — don’t just keep tuning.

In one breath

• Tuning searches hyperparameter space — but the search itself uses data, so it must never touch the final test set.
• Grid tries every combination; random samples and usually wins for the same budget, because few hyperparameters truly matter; Bayesian (Optuna) learns from past trials and pulls ahead once the space is large.
• Tune inside a Pipeline with cross-validation, so each fold preprocesses cleanly and no leakage creeps in.
• Sample log-uniform for anything spanning orders of magnitude — learning_rate, C, regularization weights.
• The CV score you tuned on is optimistic. For an honest number, evaluate once on a held-out set, or use nested CV.

Quick check