The One-Line Summary: Ten posts on boosting, every number in them produced by code that actually ran — and the six results worth keeping are that the learning rate and the tree count are one parameter, that unlimited depth is worse than stumps, that boosting's optimum was 19 trees and not 3,000, that coarser histogram binning won, that naive target encoding turned a signal-free column into a test loss of 2.7975, and that XGBoost and LightGBM are a tie nobody wants to admit.

The Arc

The Boosting series is finished. It started with a tutor who only reviews the questions you got wrong and ended with a tuning sheet, and the useful thing about writing it in order is that each post kept contradicting something I believed when I started the previous one.

Here is the whole arc, with the one result from each that survived.

AdaBoost: The Tutor Who Only Reviews the Questions You Got Wrong

Where boosting begins: reweight the examples you got wrong, fit the next learner to those, repeat. The parable is a tutor who stops re-teaching what you already know. The mechanism is exponential loss, and it is the reason AdaBoost is so sensitive to a mislabelled row — a permanently wrong answer gets permanently more attention.

Gradient Boosting: The Sculptor Who Fixes the Statue by Carving Only Its Mistakes

The generalisation: stop reweighting examples and start fitting the residual directly, which turns boosting into gradient descent in function space. The from-scratch implementation matched scikit-learn to 0.970898 against 0.970890 — five decimal places from thirty lines of NumPy, which is the clearest evidence I know that the library is not doing anything mysterious.

learning_rate and n_estimators Are One Parameter, Not Two

Sweep the learning rate, early-stop for the tree count each time, and the product barely moves: 4.50, 4.40, 4.41 across a tenfold range of learning rates. Halve the rate and you need twice the trees. This is the single result that has saved me the most compute, because it removes an entire axis from every grid search.

The Boosting Family Decision Tree

Which algorithm, on one page. Worth revisiting now that the series is done, because two of its recommendations got contradicted by later measurements.

XGBoost: The Accountant Who Read the Tax Code So Carefully He Found Money Nobody Else Saw

Second-order gradients, pruning written into the objective, and missing values as a learned direction. The result that landed hardest: second order reached 0.3322 in 50 rounds while first order sat at 0.3436 after 200 and never caught up. From scratch it agreed with the library at 0.931456 against 0.931628.

LightGBM: The Librarian Who Sorted a Million Books by Ignoring Most of Them

Histogram binning and leaf-wise growth. The finding I did not expect: max_bin=15 beat max_bin=1023, 0.1598 against 0.1619. More resolution in the split search bought variance, not accuracy. This is the post that made me stop trusting defaults that sound generous.

CatBoost: The Interpreter Who Refused to Peek at Tomorrow's Newspaper

Target leakage, measured. On a column containing literally no signal, naive target encoding produced a test loss of 2.7975; out-of-fold encoding 0.5823; deleting the column entirely 0.5464. Dropping a useless column beat every clever encoding of it, which is the most uncomfortable result in the series.

Bagging vs Boosting: The Orchestra and the Relay Team

The settled comparison. Removing one tree from a hundred costs bagging 0.0014 and costs boosting 0.2361 at the front of the chain — the crowd is interchangeable, the argument is not. Also: boosting's optimum was 19 trees at 36.1841, and running to 3,000 degraded it to 48.4704. And unlimited depth took boosting to 7.1105, worse than stumps.

I Gave XGBoost and LightGBM the Same 30 Seconds

0.98845 against 0.98825. Two ten-thousandths of an AUC point. Giving LightGBM its speed advantage back as extra search time bought nothing. The libraries differ on training and inference speed, not accuracy, and almost every comparison you will read online reports a gap smaller than its own noise.

The Boosting Tuning Cheat Sheet

All of the above, compressed into a tuning order you can follow without re-reading anything.

The Six That Changed How I Work

1. learning_rate × n_estimators is one parameter. 4.50, 4.40, 4.41.
2. max_depth has a low ceiling. Unlimited depth reached 7.1105 — worse than stumps.
3. Boosting needs early stopping. Optimum 19 trees, not 3,000; 36.1841 against 48.4704.
4. Coarser can beat finer. 0.1598 against 0.1619 for max_bin 15 versus 1023.
5. Fix the encoder before the model. 2.7975 naive, 0.5823 out-of-fold, 0.5464 for deleting the column.
6. Library choice is a tie. 0.98845 against 0.98825 — pick on speed, not accuracy.

What Confused People

Nothing, and I am going to be straight about why rather than invent a reader question.

This series drew zero substantive comments. Two spam replies and one thread I had already answered, across ten posts. That is not a mystery — it is a reach problem I measured rather than guessed at, and the diagnosis was that #datascience had a median of zero reactions across the twenty best posts of its week while #ai had fifty-three, and that I had been spending a quarter of every post's tag budget on a dead slot. That changed from the Bagging vs Boosting post onward.

I mention it because a recap that manufactures "great question from a reader!" is doing the same thing as a benchmark with no error bars. If the questions had been there I would quote them. They were not, so here is the honest version: the series was written into a very quiet room, and the fix was distributional rather than editorial.

The One-Sentence Summary

Ten posts of boosting reduce to one habit — treat the learning rate and the tree count as a single parameter, keep the trees shallow, stop early, and spend the time you save auditing whether your labels and your encoders deserve the trust you are placing in them, because every result in this series that surprised me came from the data and not from the algorithm.

What's Next?

The Boosting arc is closed. Next up is a new series on the ensembles that sit outside the bagging/boosting split:

1. Stacking — what to do when you have several good models and no idea which to trust.
2. Blending and voting — the two ways a committee can agree, and why one of them cheats.
3. Grid search vs random search — the tourist with a map versus the one who wanders.
4. Bayesian optimization — the sommelier who learns your palate in six glasses.

Follow me for the Ensembles Beyond Bagging series!

Let's Connect!

If any single number in this series changed something you do, drop a heart!

Questions? Ask in the comments — I read and respond to every one, and after the paragraph above you can see I would genuinely like the chance.

Which of the six surprised you most? Mine was the max_bin one, because I had spent two years assuming finer binning was a free accuracy knob you traded memory for. 📉

Writing a series in publication order rather than drafting it all up front turned out to be the useful constraint. Four of the ten posts contradicted something I had asserted in an earlier one, and I only noticed because the code had to run again each time. A series drafted in one sitting would have been internally consistent and wrong in at least four places.

If you're starting a boosting project this week, read the cheat sheet and skip the rest.