The One-Line Summary: Every number on this page was measured in an earlier post in this series rather than copied from a blog — the tuning order that falls out of them is
learning_rate×n_estimatorsfirst as one parameter, thenmax_deptharound 3 to 6, then stop, because unlimited depth took boosting from 2.0196 to 7.1105 test MSE and running to 3,000 trees when the optimum was 19 made it a third worse.
The Order
Nobody tunes eleven parameters. Here is the order the measurements justify.
BOOSTING TUNING ORDER
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1. learning_rate x n_estimators ← ONE parameter
pick lr, then early-stop for the tree count.
Their product is roughly conserved.
2. max_depth (or num_leaves) ← 3 to 6
the single biggest lever after step 1.
3. subsample / colsample ← 0.6 to 1.0
mild regulariser, cheap to sweep.
4. everything else ← usually noise
stop here unless you have a validation
curve telling you otherwise.
1. learning_rate and n_estimators Are One Parameter
Measured on classification, sweeping the learning rate and early-stopping for the tree count each time. The product lr × trees barely moves:
lr peak trees lr x trees
0.3 15 4.50
0.1 44 4.40
0.03 147 4.41
Three learning rates spanning a factor of ten, and the product lands at 4.50, 4.40, 4.41. Halve the learning rate and you need roughly twice the trees for the same model. So do not grid-search both — fix one, early-stop the other.
2. max_depth Has a Ceiling, and It Is Low
The mirror image of bagging. Test MSE at 100 estimators, same data:
max_depth bagging boosting
1 19.5944 5.0660
5 5.6241 2.0196
None 3.6156 7.1105
Bagging improves all the way to unlimited depth. Boosting is U-shaped and unlimited depth is worse than stumps — 7.1105 against 5.0660. A fully grown tree eats the whole residual on the first pass and leaves nothing for the rest of the chain, noise included.
Buy capacity with more trees, not deeper ones.
3. More Trees Is Not Free
Bagging and boosting answer this differently, and it is the most common way people waste a weekend:
n_estimators bagging boosting
10 39.5639 37.1438
100 36.5054 43.4739
3000 35.5388 48.4704
Bagging is monotone — 3,000 trees was its best. Boosting's optimum was 19 trees at 36.1841, and by 3,000 it had degraded to 48.4704. A third worse than its own best, from doing more of the thing that was working.
n_estimators is a smoothness knob for bagging and a capacity knob for boosting. Only one of them needs early stopping.
4. The Library-Specific Knobs
LightGBM measured effect
max_bin 15 beat 1023 (0.1598 vs 0.1619)
growth leaf-wise 0.1753 vs depth-wise 0.1784
num_leaves the dangerous default; couple it
to max_depth or it silently overfits
CatBoost measured effect
ordered TS naive target encoding: test 2.7975
out-of-fold: test 0.5823
drop the column: test 0.5464
→ on a signal-free column, deleting
it beat every encoding of it
The max_bin result is worth pausing on: the coarser binning won. More resolution in the split search is not more accuracy, it is more variance.
5. Which Library — and Why It Barely Matters
Same data, same tuning budget, 15 trials each:
test AUC tune refit predict
XGBoost 0.98845 10.8s 1.27s 17ms
LightGBM 0.98825 4.9s 0.38s 38ms
Two ten-thousandths of an AUC point. Give LightGBM its speed advantage back as extra search time and it fitted 59 candidates to XGBoost's 37 — final scores 0.98977 and 0.98990. The extra search bought nothing.
So pick on the loop you are constrained by. LightGBM wins development (3.3× faster to fit). XGBoost wins production (2.2× faster to score).
6. When Tuning Is the Wrong Job
BEFORE YOU TUNE, CHECK THESE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✗ Labels are noisy
30% flipped labels cost bagging 0.1133 accuracy
and boosting 0.1477. Boosting pays more for bad
labels than any hyperparameter can win back.
✗ A high-cardinality column is target-encoded
by hand. Fix the encoder, not the depth.
✗ Your CV and your holdout disagree by more than
the gap you are chasing. Tuning against a
leaky score optimises the leak.
✓ Clean labels, honest validation, and you still
want the last two points. Now tune.
Key Takeaways
learning_rate×n_estimatorsis one parameter — measured at 4.50, 4.40 and 4.41 across a tenfold range of learning rates.max_depth3–6, never unlimited — unlimited took boosting from 2.0196 to 7.1105, worse than stumps at 5.0660.Boosting needs early stopping and bagging does not — optimum at 19 trees versus 3,000, and 48.4704 versus 36.1841 for ignoring it.
Coarser can win — LightGBM's
max_bin=15beatmax_bin=1023at 0.1598 against 0.1619.Library choice is a tie on accuracy — 0.98845 against 0.98825. Choose on training versus inference speed instead.
Fix the data before the knobs — noisy labels cost boosting 0.1477 of accuracy, which no amount of tuning recovers.
The One-Sentence Summary
Tune the learning rate and tree count as a single parameter with early stopping, keep max_depth between 3 and 6 because unlimited depth is worse than stumps, sweep subsample if you are bored, and spend everything you save on checking whether your labels and your validation split deserve the trust you are placing in them.
What's Next?
- The series recap — the whole Boosting arc in one page, with the numbers that survived.
- Stacking — what to do when you have several good models and no idea which to trust.
- Blending and voting — the two ways a committee can agree, and why one of them cheats.
- Grid search vs random search — and why the tourist with a map loses.
Follow me for the next article in the Boosting: The Complete Guide series!
Let's Connect!
If this saved you a grid search, drop a heart!
Questions? Ask in the comments — I read and respond to every one.
What's the hyperparameter you've stopped tuning entirely? Mine is n_estimators on random forests — I set it to 500 and never think about it again, because the measurement says it cannot hurt me. 🔧
The reason cheat sheets are usually wrong is that they are assembled from other cheat sheets. Every figure on this page came from code that ran in an earlier post in this series, which is a low bar that most tuning advice does not clear — and the two most useful lines here, that coarser binning won and that deleting a column beat encoding it, are both things I would not have believed from a table someone else wrote.
Bookmark this before your next grid search, and check the depth row first.
