The One-Line Summary: Bagging and boosting both turn mediocre trees into strong models, from exactly opposite directions — the orchestra plays every part at once and averages away the wobble, the relay hands each runner only the distance the previous one failed to cover. Measured on identical data they finish within 0.08 of the same total error, and yet removing a single tree costs the orchestra 0.0014 and costs the relay 0.2361.
The Parable of the Amberton Festival
Every autumn Amberton holds two contests on the same field: the orchestra plays at noon, and the relay runs at four. For thirty years the town treated them as unrelated amusements. Then a bandmaster who also kept the running club's stopwatch noticed that both were answers to the same question — how do you get a hundred imperfect people to produce one correct result? — and that the two answers were exact opposites.
The Orchestra: Forty Players, One Note
Not one of the forty violinists is perfectly in tune. Ask any of them to play alone and you will hear it. Some run a little sharp, some a little flat, and no two are wrong in the same way on the same evening.
Played together, something strange happens. The sharpness of one player sits on top of the flatness of another and the two quietly cancel. Nobody is corrected. Nobody listens to anybody else. They simply all play at once, and the hall hears a note steadier than any single instrument producing it.
THE ORCHESTRA — FORTY PLAYERS, ONE NOTE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The note on the page: 440
violin 1 plays 443 (+3 sharp)
violin 2 plays 437 (-3 flat)
violin 3 plays 441 (+1 sharp)
violin 4 plays 436 (-4 flat)
...
violin 40 plays 442 (+2 sharp)
what the hall hears: 440.1
No player was corrected. No player heard another.
Each was wrong; the wrongness pointed in random
directions and cancelled.
Add a 41st violin and the hall gets steadier still.
It can never get worse.
This is called VARIANCE REDUCTION.
The important detail is the one that sounds like a limitation: no violinist is trying to fix anyone. The cancellation is a free side effect of independence.
The Relay: Four Runners, One Race
The relay is built on the opposite principle. Nobody runs the whole race.
The first runner covers most of the distance, badly. The second does not start at the beginning — he starts exactly where the first one stopped, and covers most of what is left. The third takes the remainder of that. Each runner's job is defined entirely by the failure of the runner before.
"Why should the second man repeat the first man's ground? Give him only the distance the first man failed to cover."
THE RELAY — EACH RUNNER TAKES WHAT IS LEFT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Distance to the line: 1000 m
runner 1 covers 700 → 300 left
runner 2 covers 210 → 90 left
runner 3 covers 63 → 27 left
runner 4 covers 19 → 8 left
Runner 2 never attempts the first 700 m. That part
is finished. He is handed 300 m of pure failure and
nothing else.
Reorder them and it is not the same race — runner
4's leg exists only because runners 1-3 already ran.
This is called FITTING THE RESIDUAL.
Four runners got within 8 m. Forty violinists would still be averaging.
The Two Years It Went Wrong
Amberton remembers two festivals badly, and each one broke a different method.
In one year the orchestra tuned from a piano that had been left by a window all winter and had gone flat. All forty players tuned to it faithfully. All forty played the same wrong note, and the average of forty identical mistakes is that mistake. A forty-first violin would not have helped.
In another year a groundsman set the first marker forty metres short. Runner one stopped where the marker said. Runner two measured his leg from that point, honestly and carefully, and so did three and four. Every correction after the first was faithful to a false premise, and the team ran, confidently and in perfect form, to the wrong place.
TWO WAYS TO FAIL
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
THE FLAT PIANO
Every violin tuned from the same flat piano.
All forty played 436. The average was 436.
A 41st violin also played 436.
✗ Averaging cannot remove an error that everyone
shares. This is CORRELATION, and it is the
floor the orchestra can never get under.
THE SHORT MARKER
The first marker was set 40 m short.
Runner 2 measured his leg from it. So did 3 and 4.
Each correction was faithful to a false start.
✗ The relay never questions the baton. It
inherits it. This is NOISE SENSITIVITY.
Why Each One Works
THE MATHEMATICS OF AVERAGING VS CORRECTING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
AVERAGING N members whose errors have variance σ²
and average pairwise correlation ρ:
variance = ρσ² + (1-ρ)σ²/N
↑ ↑
shared error private error
never shrinks → 0 as N grows
REQUIREMENTS
1. Each member is fit independently.
2. Members must DISAGREE (low ρ), or step 1
bought you nothing.
3. Each member should be LOW BIAS — averaging
removes variance and never touches bias.
CORRECTING: member m is fit to what m-1 left over.
F_m(x) = F_{m-1}(x) + lr · h_m(x)
REQUIREMENTS
1. Order is fixed. Members are not swappable.
2. Each member should be WEAK — a strong one
eats the whole residual and the chain stops.
3. There is no ρ floor to worry about. There is
also no protection from a wrong target.
Two methods, two entirely different failure modes, and — this is the part almost nobody checks — very often the same final accuracy.
What Are Bagging and Boosting?
Bagging (bootstrap aggregating) fits copies of the same learner on different bootstrap resamples of the training set, then averages their predictions. The members never see each other. The fit is embarrassingly parallel.
Boosting fits members in sequence. Each one is trained on the errors the current ensemble still makes, and is added to the ensemble with a small weight. The fit is inherently serial.
The Math
For bagging, the variance of the average of predictors with individual variance and mean pairwise correlation is:
The second term vanishes as . The first does not. Everything interesting about bagging is a fight to lower , which is exactly what random forests do by sampling features at each split.
For boosting, the model is a stagewise additive expansion. At stage we fit a new learner to the negative gradient of the loss and take a damped step:
There is no in that expression, because independence was never the point.
Step by Step
Bagging
- Draw a bootstrap sample of size with replacement.
- Fit a deep, unpruned learner on it.
- Repeat times, in any order, on any number of machines.
- Average (regression) or vote (classification).
Boosting
- Start with a constant prediction, usually the mean.
- Compute what the current ensemble still gets wrong.
- Fit a small, deliberately weak learner to that.
- Add it, scaled by the learning rate. Return to step 2.
The Decomposition That Settles It
The textbook line is "bagging reduces variance, boosting reduces bias." That is a claim about numbers, so let us get the numbers. Sixty independent training sets, the same test set, and a decomposition of the error of four models into the part that is systematic and the part that is jitter.
import numpy as np
from sklearn.datasets import make_friedman1
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor, GradientBoostingRegressor
N_FITS, N_TRAIN, N_TEST = 60, 300, 500
X_test, y_true = make_friedman1(n_samples=N_TEST, noise=0.0, random_state=7)
def decompose(make_model, label):
preds = np.zeros((N_FITS, N_TEST))
for i in range(N_FITS):
Xtr, ytr = make_friedman1(n_samples=N_TRAIN, noise=1.0,
random_state=1000 + i)
preds[i] = make_model(i).fit(Xtr, ytr).predict(X_test)
mean_pred = preds.mean(axis=0)
bias2 = float(np.mean((mean_pred - y_true) ** 2))
var = float(np.mean(preds.var(axis=0)))
mse = float(np.mean((preds - y_true) ** 2))
print(f"{label:<26} bias2 {bias2:7.4f} "
f"variance {var:7.4f} total {mse:7.4f}")
print("BIAS-VARIANCE DECOMPOSITION")
print("60 independent training sets, Friedman #1")
print("-" * 58)
decompose(lambda i: DecisionTreeRegressor(max_depth=None, random_state=42),
"single deep tree")
decompose(lambda i: BaggingRegressor(DecisionTreeRegressor(max_depth=None),
n_estimators=100, random_state=42, n_jobs=-1), "bagging, 100 deep trees")
decompose(lambda i: GradientBoostingRegressor(n_estimators=100, max_depth=1,
learning_rate=0.1, random_state=42), "boosting, 100 stumps")
decompose(lambda i: DecisionTreeRegressor(max_depth=1, random_state=42),
"single stump")
BIAS-VARIANCE DECOMPOSITION
60 independent training sets, Friedman #1
----------------------------------------------------------
single deep tree bias2 3.2389 variance 8.1118 total 11.3507
bagging, 100 deep trees bias2 3.6628 variance 1.0594 total 4.7222
boosting, 100 stumps bias2 4.1052 variance 0.5443 total 4.6495
single stump bias2 16.1720 variance 1.9015 total 18.0735
Read each ensemble against its own base learner, which is the comparison people usually skip.
Bagging took a deep tree and cut its variance from 8.1118 to 1.0594 — a factor of 7.7 — while bias went slightly up, from 3.2389 to 3.6628. That is the orchestra exactly: it did nothing whatsoever about the note being wrong, and everything about the wobble.
Boosting took a stump and cut its bias from 16.1720 to 4.1052, a factor of 3.9. That is the relay: a runner who could only ever cover a fraction of the track, turned into a team that reaches the line.
And here is the part the textbook line gets wrong. Boosting also cut variance, from 1.9015 to 0.5443, by a factor of 3.5. Shrinkage and weak learners are a regularizer, not just a bias fix. "Boosting reduces bias" is directionally right and quantitatively incomplete.
Then look at the two totals: 4.7222 and 4.6495. Two philosophies, opposite mechanisms, opposite failure modes, and they arrive within 0.08 of each other.
Are the Members Interchangeable?
The parable claims the orchestra's players are swappable and the relay's runners are not. That is testable. Build both from scratch, confirm they match the library, then remove exactly one tree from a hundred and see what it costs.
import numpy as np
from sklearn.datasets import make_friedman1
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor, GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
class MyBagging:
"""Resample, fit deep trees independently, average."""
def __init__(self, n=100, seed=42):
self.n, self.seed = n, seed
def fit(self, X, y):
rng = np.random.RandomState(self.seed)
self.trees_ = []
for _ in range(self.n):
idx = rng.randint(0, len(X), len(X)) # bootstrap
t = DecisionTreeRegressor(max_depth=None, random_state=0)
self.trees_.append(t.fit(X[idx], y[idx]))
return self
def predict(self, X):
return np.mean([t.predict(X) for t in self.trees_], axis=0)
class MyBoosting:
"""Fit each tree to what is LEFT OVER."""
def __init__(self, n=100, lr=0.1, depth=3, seed=42):
self.n, self.lr, self.depth, self.seed = n, lr, depth, seed
def fit(self, X, y):
self.f0_ = y.mean()
resid = y - self.f0_
self.trees_ = []
for _ in range(self.n):
t = DecisionTreeRegressor(max_depth=self.depth,
random_state=self.seed)
t.fit(X, resid)
resid = resid - self.lr * t.predict(X) # still wrong
self.trees_.append(t)
return self
def predict(self, X):
return self.f0_ + self.lr * np.sum(
[t.predict(X) for t in self.trees_], axis=0)
X, y = make_friedman1(n_samples=2000, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
bag = MyBagging(100).fit(Xtr, ytr)
boo = MyBoosting(100, 0.1, 3).fit(Xtr, ytr)
skl_b = BaggingRegressor(DecisionTreeRegressor(), n_estimators=100,
random_state=42, n_jobs=-1).fit(Xtr, ytr)
skl_g = GradientBoostingRegressor(n_estimators=100, max_depth=3,
learning_rate=0.1, random_state=42).fit(Xtr, ytr)
print("FROM SCRATCH vs SCIKIT-LEARN (test MSE)")
print("-" * 52)
print(f" bagging mine {mean_squared_error(yte, bag.predict(Xte)):8.4f}"
f" sklearn {mean_squared_error(yte, skl_b.predict(Xte)):8.4f}")
print(f" boosting mine {mean_squared_error(yte, boo.predict(Xte)):8.4f}"
f" sklearn {mean_squared_error(yte, skl_g.predict(Xte)):8.4f}")
def bag_pred(trees): return np.mean([t.predict(Xte) for t in trees], axis=0)
def boo_pred(trees): return boo.f0_ + boo.lr * np.sum(
[t.predict(Xte) for t in trees], axis=0)
full_b = mean_squared_error(yte, bag_pred(bag.trees_))
full_g = mean_squared_error(yte, boo_pred(boo.trees_))
print("\nREMOVING ONE MEMBER FROM 100 (test MSE)")
print("-" * 52)
print(f"{'':<22}{'bagging':>12}{'boosting':>12}")
print(f"{'all 100':<22}{full_b:>12.4f}{full_g:>12.4f}")
for name, keep in [("drop the 1st tree", list(range(1, 100))),
("drop the 50th tree", [i for i in range(100) if i != 49]),
("drop the 100th tree", list(range(0, 99)))]:
print(f"{name:<22}"
f"{mean_squared_error(yte, bag_pred([bag.trees_[i] for i in keep])):>12.4f}"
f"{mean_squared_error(yte, boo_pred([boo.trees_[i] for i in keep])):>12.4f}")
FROM SCRATCH vs SCIKIT-LEARN (test MSE)
----------------------------------------------------
bagging mine 3.6307 sklearn 3.5487
boosting mine 2.0786 sklearn 2.0770
REMOVING ONE MEMBER FROM 100 (test MSE)
----------------------------------------------------
bagging boosting
all 100 3.6307 2.0786
drop the 1st tree 3.6293 2.3147
drop the 50th tree 3.6295 2.1008
drop the 100th tree 3.6310 2.0806
Thirty lines each, and both land on the library: boosting at 2.0786 against scikit-learn's 2.0770.
Now the removals. For bagging, dropping the first, the fiftieth or the hundredth tree moves test error by at most 0.0014, and dropping the last tree (3.6310) was very slightly worse than dropping the first (3.6293) — which is noise, and that is the point. The members are genuinely interchangeable. There is no first violin.
For boosting, dropping the first tree costs 0.2361 and dropping the hundredth costs 0.0020. The same operation is 118 times more expensive at the front of the chain than at the back. Tree one is running the first 700 metres. Tree one hundred is running the last eight.
One honest correction while I was building this. My first version shuffled the members and announced that boosting was "destroyed" — and printed a number identical to the unshuffled one. Of course it did: prediction is a sum, and addition commutes. Order matters while fitting, not while predicting. The removal test is the right experiment; the shuffle test was me describing a result I expected instead of reading the one I got.
The Mirror Image: How Deep Should the Trees Be?
If bagging fixes variance and boosting fixes bias, then they should want opposite base learners. Bagging should want deep trees — low bias, high variance, exactly the raw material averaging can improve. Boosting should want shallow ones — leave it something to correct.
import numpy as np
from sklearn.datasets import make_friedman1
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor, GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
X, y = make_friedman1(n_samples=2000, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
print("BASE-LEARNER DEPTH")
print("test MSE, 100 estimators each, mean of 3 seeds")
print("-" * 50)
print(f"{'max_depth':>10} {'bagging':>10} {'boosting':>10}")
for d in [1, 2, 3, 5, 8, None]:
bag, boo = [], []
for s in range(3):
b = BaggingRegressor(DecisionTreeRegressor(max_depth=d),
n_estimators=100, random_state=s, n_jobs=-1).fit(Xtr, ytr)
g = GradientBoostingRegressor(n_estimators=100, max_depth=d,
learning_rate=0.1, random_state=s).fit(Xtr, ytr)
bag.append(mean_squared_error(yte, b.predict(Xte)))
boo.append(mean_squared_error(yte, g.predict(Xte)))
print(f"{str(d):>10} {np.mean(bag):>10.4f} {np.mean(boo):>10.4f}")
BASE-LEARNER DEPTH
test MSE, 100 estimators each, mean of 3 seeds
--------------------------------------------------
max_depth bagging boosting
1 19.5944 5.0660
2 13.2544 2.6711
3 8.9423 2.0769
5 5.6241 2.0196
8 3.9657 3.0942
None 3.6156 7.1105
Two curves running in opposite directions.
Bagging improves monotonically with depth and is best at unlimited: 19.5944 with stumps, 3.6156 with fully grown trees. Give the orchestra players who can actually play; averaging will handle the wobble.
Boosting is U-shaped. Best at depth 5 (2.0196), and unlimited depth is a disaster at 7.1105 — worse than boosting stumps. A fully grown tree consumes the entire residual on the first pass, leaves nothing for anyone else, and what it consumed included the noise.
The practical rule falls straight out: if you are bagging, stop pruning. If you are boosting, keep the trees small and buy your capacity with more of them instead.
Where Boosting Breaks
Two failure modes, both predicted by the parable, both measurable.
import numpy as np
from sklearn.datasets import make_classification, make_friedman1
from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier,
BaggingRegressor, GradientBoostingRegressor)
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
X, y = make_classification(n_samples=2000, n_features=20, n_informative=8,
n_redundant=2, flip_y=0.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
print("LABEL-NOISE SENSITIVITY (test accuracy, mean of 5 seeds)")
print("-" * 64)
print(f"{'flipped':>8} {'bagging (RF)':>14} {'boosting (GB)':>15} {'gap':>7}")
rows = []
for frac in [0.00, 0.05, 0.10, 0.20, 0.30]:
rf_s, gb_s = [], []
for seed in range(5):
rng = np.random.RandomState(seed)
y_noisy = ytr.copy()
n_flip = int(frac * len(y_noisy))
if n_flip:
idx = rng.choice(len(y_noisy), n_flip, replace=False)
y_noisy[idx] = 1 - y_noisy[idx]
rf_s.append(RandomForestClassifier(n_estimators=200, random_state=seed,
n_jobs=-1).fit(Xtr, y_noisy).score(Xte, yte))
gb_s.append(GradientBoostingClassifier(n_estimators=200, max_depth=3,
random_state=seed).fit(Xtr, y_noisy).score(Xte, yte))
rf_m, gb_m = np.mean(rf_s), np.mean(gb_s)
rows.append((rf_m, gb_m))
print(f"{frac:>7.0%} {rf_m:>14.4f} {gb_m:>15.4f} {rf_m-gb_m:>+7.4f}")
print(f"\nclean -> 30% noise: bagging loses {rows[0][0]-rows[-1][0]:.4f},"
f" boosting loses {rows[0][1]-rows[-1][1]:.4f}")
# Does adding more members ever hurt?
X, y = make_friedman1(n_samples=400, noise=5.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=42)
print("\nADDING MORE ESTIMATORS (small noisy sample)")
print("-" * 56)
print(f"{'n_estimators':>13} {'bagging':>10} {'boosting':>10}")
for n in [10, 100, 1000, 3000]:
b = BaggingRegressor(DecisionTreeRegressor(), n_estimators=n,
random_state=42, n_jobs=-1).fit(Xtr, ytr)
g = GradientBoostingRegressor(n_estimators=n, max_depth=3,
learning_rate=0.1, random_state=42).fit(Xtr, ytr)
print(f"{n:>13} {mean_squared_error(yte, b.predict(Xte)):>10.4f}"
f" {mean_squared_error(yte, g.predict(Xte)):>10.4f}")
g = GradientBoostingRegressor(n_estimators=3000, max_depth=3, learning_rate=0.1,
random_state=42).fit(Xtr, ytr)
errs = [mean_squared_error(yte, p) for p in g.staged_predict(Xte)]
print(f"\nboosting best at n={int(np.argmin(errs))+1} (MSE {min(errs):.4f}),"
f" ends at n=3000 (MSE {errs[-1]:.4f})")
LABEL-NOISE SENSITIVITY (test accuracy, mean of 5 seeds)
----------------------------------------------------------------
flipped bagging (RF) boosting (GB) gap
0% 0.8560 0.8317 +0.0243
5% 0.8480 0.8113 +0.0367
10% 0.8350 0.8067 +0.0283
20% 0.8013 0.7467 +0.0547
30% 0.7427 0.6840 +0.0587
clean -> 30% noise: bagging loses 0.1133, boosting loses 0.1477
ADDING MORE ESTIMATORS (small noisy sample)
--------------------------------------------------------
n_estimators bagging boosting
10 39.5639 37.1438
100 36.5054 43.4739
1000 35.7662 48.4598
3000 35.5388 48.4704
boosting best at n=19 (MSE 36.1841), ends at n=3000 (MSE 48.4704)
Noise. Going from clean labels to 30% flipped, bagging gives up 0.1133 of accuracy and boosting gives up 0.1477. The gap between them widens from +0.0243 to +0.0587. A mislabelled row is a residual that never goes away, so boosting keeps sending runners at it. Bagging simply averages over it. This is the short marker, and it is why boosting is the wrong first choice on hand-labelled data.
More members. Bagging on a small noisy sample goes 39.5639 → 35.5388 as it grows from 10 to 3,000 trees. Monotone, and it never gets worse, exactly as the variance formula promises. Boosting starts at 37.1438 and climbs to 48.4704. Its best model was 19 trees; running it to 3,000 made it a third worse than its own optimum.
That asymmetry is the single most useful thing on this page. n_estimators is a capacity knob for boosting and a smoothness knob for bagging. One of them needs early stopping and a validation set. The other you can set to "more" and walk away.
When Does Each Help Most?
| Bagging | Boosting | |
|---|---|---|
| Attacks | Variance | Bias (and some variance) |
| Base learner | Deep, unpruned | Shallow, depth 3–6 |
| Fitting | Parallel | Serial |
| More members | Never hurts | Overfits eventually |
| Noisy labels | Degrades gracefully | Chases the noise |
| Needs early stopping | No | Yes |
| Tuning effort | Almost none | Real |
| Typical accuracy, clean tabular | Good | Usually better |
Reach for bagging when labels are noisy or hand-made, when you want a defensible baseline in one line, when you have cores and no time, or when nobody will be around to babysit a validation curve.
Reach for boosting when labels are trustworthy, when the last two points of accuracy are worth real tuning effort, and when you can hold out a validation set to stop on.
Bagging and Boosting Hyperparameters
| Parameter | Bagging | Boosting |
|---|---|---|
n_estimators |
100–1000, more is safe | tune with early stopping |
max_depth |
None |
3–6 |
learning_rate |
n/a | 0.01–0.1, lower needs more trees |
max_features |
sqrt lowers ρ |
mild regularizer |
subsample |
bootstrap is the mechanism | 0.5–0.8 adds randomness |
| What to tune first | almost nothing |
learning_rate × n_estimators
|
Quick Reference Card
BAGGING vs BOOSTING: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT THEY ARE:
Bagging — fit members independently, average them.
Boosting — fit each member to the leftover error.
THE FORMULAS:
bagging Var = ρσ² + (1-ρ)σ²/N
boosting F_m = F_{m-1} + lr · h_m
MEASURED HERE (Friedman #1):
deep tree bias 3.2389 var 8.1118
bagged bias 3.6628 var 1.0594 (var ÷7.7)
stump bias 16.1720 var 1.9015
boosted bias 4.1052 var 0.5443 (bias ÷3.9)
DROP ONE TREE OF 100:
bagging costs 0.0014 boosting costs 0.2361
WHEN TO USE:
✓ bagging — noisy labels, no time, no babysitting
✓ boosting — clean labels, tuning budget, val set
WHEN NOT EFFECTIVE:
✗ bagging — members all wrong the same way (high ρ)
✗ boosting — mislabelled rows, deep trees, no early stop
SKLEARN:
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import GradientBoostingRegressor
Key Takeaways
They attack different halves of the error — bagging cut a deep tree's variance from 8.1118 to 1.0594 while nudging bias slightly up; boosting cut a stump's bias from 16.1720 to 4.1052.
"Boosting reduces bias" is incomplete — boosting also cut variance by a factor of 3.5, from 1.9015 to 0.5443. Shrinkage and weak learners regularize.
They arrive at nearly the same place — 4.7222 against 4.6495 total error, from opposite directions.
Bagging's members are interchangeable, boosting's are a chain — removing one tree of a hundred costs bagging at most 0.0014, and costs boosting 0.2361 at the front versus 0.0020 at the back.
They want opposite base learners — bagging is best with unlimited depth (3.6156), boosting is best at depth 5 (2.0196) and is worse than useless at unlimited depth (7.1105).
n_estimatorsmeans two different things — bagging improved all the way to 3,000 trees; boosting's optimum was 19 and by 3,000 it was a third worse.Boosting pays more for bad labels — 30% flipped labels cost bagging 0.1133 accuracy and boosting 0.1477, with the gap widening monotonically.
Compare each ensemble to its own base learner — against each other the two totals look interchangeable, and the entire mechanism is invisible.
The One-Sentence Summary
Bagging and boosting are the orchestra and the relay — forty players sounding one note by cancelling each other's independent mistakes, against four runners each handed only the distance the last one failed to cover — and the measurements say the orchestra divides variance by 7.7 while barely touching bias, the relay divides bias by 3.9, they finish within 0.08 of the same total error, and the only way to tell them apart from the outside is to remove one member and watch it cost the orchestra 0.0014 and the relay 0.2361.
What's Next?
Now that you can tell the two families apart, you're ready for:
- XGBoost vs LightGBM in 10 minutes — tomorrow. Same data, same tuning budget, honest numbers.
- The hyperparameter tuning cheat sheet — everything that matters, on one page.
- Stacking — what to do when you have several good models and no idea which to trust.
- Blending and voting — the two ways a committee can agree, and why one of them cheats.
Follow me for the next article in the Boosting: The Complete Guide series!
Let's Connect!
If the orchestra and the relay made ensembles click, drop a heart!
Questions? Ask in the comments — I read and respond to every one.
Which one do you reach for first? I default to bagging for the baseline and only spend the tuning budget on boosting once I've seen how clean the labels are — because a random forest has never once surprised me in production, and gradient boosting has. ⚖️
What strikes me about these two, after measuring them properly, is that the family argument was never about accuracy — they land in the same place. It is about what you are willing to be responsible for. Bagging asks for independence and gives you a model that cannot be made worse by adding to it. Boosting asks for attention and gives you a model that will follow you anywhere, including off a cliff, because every tree after the first is an act of faith in the ones before it. One is a crowd, the other is an argument. The crowd is safer; the argument is usually right; and the engineering judgment is knowing which of those you need this week.
Forward this to the teammate who set n_estimators=5000 on a gradient booster and went to lunch.
