The One-Line Summary: Replacing a category with the average target for that category is the most common feature-engineering trick in tabular machine learning and it quietly hands the model the answer — measured here, it turned a column containing literally no signal into a training loss of 0.2466 and a test loss of 2.7975, five times worse than deleting the column; CatBoost's ordered target statistics fix the leak by letting each row see only rows that came before it.

The Parable of the Court at Vashti

The court employed interpreters, because testimony arrived in six languages and the judges read one. An interpreter's job was to render each witness's words faithfully, and for two centuries the court considered this a solved problem.

Then someone noticed that the interpreters were extraordinarily good at old cases and merely average at new ones.

The Old Way: Read the Whole File First

Every interpreter was handed the complete case file before beginning. This was considered basic professionalism — how could you translate testimony about a boundary dispute without knowing it was a boundary dispute?

The file included the verdict.

WHY THE OLD INTERPRETERS LOOKED SO GOOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Witness says a word that could mean
"he took it" or "he was given it".

Interpreter has read the file. The file says
GUILTY.

She writes "he took it."

She is not lying. She is not even conscious of
choosing. The ambiguity resolved itself the
moment she knew how it ended.

On closed cases her renderings are uncanny.
On open cases she is ordinary, and nobody can
work out why.

The interpreters were not corrupt. They were contaminated, which is worse, because contamination leaves no one to blame and nothing obvious to fix.

The Investigation

A clerk named Ilaya was asked to audit the interpreters and did something nobody had tried: she took a hundred old cases, stripped the verdicts, and had them re-translated by the same people.

THE SAME INTERPRETERS, VERDICTS HIDDEN
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
with the file (as always)   near-perfect renderings
without the verdict          ordinary renderings
                             indistinguishable from
                             a first-year apprentice

The skill everyone had been admiring for two
centuries was not skill. It was the verdict,
travelling backwards into the testimony.

The court's instinct was to ban case files entirely. Ilaya argued against it: context genuinely helps, and an interpreter working blind is worse than one working informed. The problem was never the file. It was which parts of the file.

"Let her read everything that was written before the words she is translating. Nothing that was written after. The rule is not ignorance — it is chronology."

ILAYA'S RULE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Translating testimony from the 3rd of the month?

  You may read: everything filed on the 1st, 2nd.
  You may not read: the 4th onward. Ever.
                    Especially the verdict.

Each document is translated using only the
court's knowledge AS IT STOOD at that moment.

Slower. Occasionally the interpreter has almost
no context and must simply admit it.
Those renderings are honest, and they hold up.

The first interpreters trained this way looked worse on the court's historical records. They were the first ones who did not get worse when the case was live.

What Is Target Encoding, and Why Does It Leak?

Hard pivot.

You have a categorical column with many levels — city, merchant, user ID, product SKU. One-hot encoding it produces thousands of columns. The popular alternative is target encoding: replace each level with the mean of the target for that level.

Look at the index set. It includes $j = i$ . Row $i$ 's own label is in the numerator of row $i$ 's feature. If a level appears once, its encoding is its label, exactly. If it appears three times, the encoding is two-thirds someone else's answer and one-third your own.

That is the verdict travelling backwards into the testimony.

The Fix, Formally

CatBoost computes ordered target statistics. Fix a random permutation $\sigma$ of the rows. For row $i$ , use only rows that precede it:

where $p$ is a prior (the global mean) and $a$ its weight. Row $i$ is excluded by construction — it hasn't happened yet. The prior does the work when history is thin.

The same principle is applied to the gradients during boosting, which is where the name "ordered boosting" comes from, but the target-statistic version is where the damage usually is.

Measuring the Leak

The cleanest possible test: build a categorical column that carries no information whatsoever, and see whether target encoding can conjure signal out of it. 6,000 rows, 2,000 levels — roughly three rows per level, which is exactly the regime where people reach for target encoding.

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split, KFold
from sklearn.metrics import log_loss
from sklearn.ensemble import HistGradientBoostingClassifier

rng = np.random.default_rng(0)
n, K = 6000, 2000
cat = rng.integers(0, K, n)              # pure noise: 2,000 levels
x1 = rng.normal(size=n)
y = (0.9 * x1 + rng.normal(0, 1.0, n) > 0).astype(int)   # y ignores cat

X = pd.DataFrame({"cat": cat, "x1": x1})
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
ctr, cte = Xtr["cat"].values, Xte["cat"].values
gm = ytr.mean()


def score(A, B):
    m = HistGradientBoostingClassifier(max_iter=200, random_state=0).fit(A, ytr)
    return (log_loss(ytr, m.predict_proba(A)[:, 1]),
            log_loss(yte, m.predict_proba(B)[:, 1]))


full_mean = pd.Series(ytr).groupby(ctr).mean()
B = Xte.copy(); B["cat"] = pd.Series(cte).map(full_mean).fillna(gm).values

A = Xtr.copy(); A["cat"] = pd.Series(ctr).map(full_mean).fillna(gm).values
tr, te = score(A, B)
print(f"  naive target encoding   train {tr:.4f}   test {te:.4f}   gap {te-tr:+.4f}")

A2 = Xtr.copy(); enc = np.full(len(Xtr), np.nan); ys = pd.Series(ytr)
for itr, iva in KFold(5, shuffle=True, random_state=0).split(Xtr):
    mm = ys.iloc[itr].groupby(ctr[itr]).mean()
    enc[iva] = pd.Series(ctr[iva]).map(mm).fillna(ys.iloc[itr].mean()).values
A2["cat"] = enc
tr, te = score(A2, B)
print(f"  out-of-fold encoding    train {tr:.4f}   test {te:.4f}   gap {te-tr:+.4f}")

tr, te = score(Xtr[["x1"]], Xte[["x1"]])
print(f"  drop the column         train {tr:.4f}   test {te:.4f}   gap {te-tr:+.4f}")

  naive target encoding   train 0.2466   test 2.7975   gap +2.5508
  out-of-fold encoding    train 0.4436   test 0.5823   gap +0.1388
  drop the column         train 0.4964   test 0.5464   gap +0.0500

Sit with the first row.

The column is random integers. It has no relationship to the target — I generated y from x1 alone. And naive target encoding produced a training loss of 0.2466, better than the honest model can achieve, and a test loss of 2.7975 — roughly five times worse than simply deleting the column.

A model that has memorised the training set and learned nothing looks, during training, exactly like a breakthrough.

Ordered Target Statistics from Scratch

Ilaya's rule is about thirty lines. Walk the rows in a random permutation, encode each one from the running totals so far, then add it to the totals.

import numpy as np


def ordered_target_stats(cat, y, prior, a=1.0, seed=0):
    """Each row is encoded using only rows earlier in the permutation."""
    rng = np.random.default_rng(seed)
    perm = rng.permutation(len(cat))
    enc = np.empty(len(cat))
    csum, ccnt = {}, {}
    for pos in perm:
        c = cat[pos]
        s, k = csum.get(c, 0.0), ccnt.get(c, 0)
        enc[pos] = (s + a * prior) / (k + a)   # history only — self excluded
        csum[c] = s + y[pos]                   # now add self, for later rows
        ccnt[c] = k + 1
    return enc

The two lines after the assignment are the whole idea. The row contributes to everyone after it and never to itself.

Run all four encodings on the same data, both when the column is noise and when it genuinely carries signal, three seeds each:

=== NO signal in the column (test logloss, 3 seeds) ===
  seed   naive    oof      ordered   drop
     0   2.7975   0.5823   0.5812   0.5464
     1   3.2548   0.5780   0.6050   0.5545
     2   2.9780   0.5988   0.6014   0.5702
  MEAN   3.0101   0.5863   0.5959   0.5570

=== REAL signal in the column (test logloss, 3 seeds) ===
  seed   naive    oof      ordered   drop
     0   2.2485   0.5707   0.6737   0.6398
     1   2.5480   0.5564   0.6239   0.6139
     2   2.0538   0.5601   0.6670   0.6518
  MEAN   2.2834   0.5624   0.6549   0.6352

Four readings, and the third one surprised me.

Naive encoding is a catastrophe either way — 3.0101 and 2.2834 against roughly 0.56 for every honest method. It does not matter whether the column has signal. The leak dominates.

When the column carries real signal, encoding it pays. Out-of-fold scored 0.5624 against 0.6352 for dropping the column. That is the case for target encoding, and it is a real one.

My ordered implementation lost to plain out-of-fold — 0.6549 against 0.5624. I expected the opposite, and I am reporting it because it is what the code printed.

Dropping a signal-free column beats every encoding of it. 0.5570 against 0.5863 out-of-fold. No scheme recovers information that was never there; encoding only adds noise with a plausible face.

Why the Ordered Version Underperformed

A result you did not expect is worth ten minutes before you publish it. The obvious suspect is variance: early rows in the permutation have almost no history, so their encodings are mostly prior. CatBoost's actual answer to this is to average several permutations — so I tried that.

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss
from sklearn.ensemble import HistGradientBoostingClassifier


def ordered_target_stats(cat, y, prior, a=1.0, seed=0):
    rng = np.random.default_rng(seed)
    perm = rng.permutation(len(cat))
    enc = np.empty(len(cat)); csum, ccnt = {}, {}
    for pos in perm:
        c = cat[pos]; s, k = csum.get(c, 0.0), ccnt.get(c, 0)
        enc[pos] = (s + a * prior) / (k + a)
        csum[c] = s + y[pos]; ccnt[c] = k + 1
    return enc


rng = np.random.default_rng(0)
n, K = 6000, 2000
cat = rng.integers(0, K, n)
eff = rng.normal(0, 1.5, K)                     # this time the column MATTERS
x1 = rng.normal(size=n)
y = (eff[cat] + 0.9 * x1 + rng.normal(0, 1.0, n) > 0).astype(int)
X = pd.DataFrame({"cat": cat, "x1": x1})
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
ctr, cte = Xtr["cat"].values, Xte["cat"].values
gm = ytr.mean()
fm = pd.Series(ytr).groupby(ctr).mean()
B = Xte.copy(); B["cat"] = pd.Series(cte).map(fm).fillna(gm).values

print("  permutations   test logloss")
for P in (1, 2, 4, 8, 16):
    encs = np.mean([ordered_target_stats(ctr, ytr, gm, seed=s)
                    for s in range(P)], axis=0)
    A = Xtr.copy(); A["cat"] = encs
    m = HistGradientBoostingClassifier(max_iter=200, random_state=0).fit(A, ytr)
    print(f"  {P:>12}   {log_loss(yte, m.predict_proba(B)[:, 1]):.4f}")

# and the diagnostic that explains the result
perm = np.random.default_rng(0).permutation(len(ctr))
seen, hist = {}, []
for pos in perm:
    c = ctr[pos]; hist.append(seen.get(c, 0)); seen[c] = seen.get(c, 0) + 1
hist = np.array(hist)
print(f"\n  rows with ZERO prior observations of their own level: "
      f"{(hist==0).sum():,} of {len(hist):,} ({(hist==0).mean()*100:.1f}%)")
print(f"  first 10% of the permutation: "
      f"{(hist[:len(hist)//10]==0).mean()*100:.1f}% have no history")
print(f"  last  10% of the permutation: "
      f"{(hist[-len(hist)//10:]==0).mean()*100:.1f}% have no history")

  permutations   test logloss
             1   0.6737
             2   0.7020
             4   0.7110
             8   0.6810
            16   0.6854

Averaging did not help. So the problem is not variance, and I went looking for what it actually is:

  rows with ZERO prior observations of their own level: 1,747 of 4,200 (41.6%)
  first 10% of the permutation: 92.1% have no history
  last  10% of the permutation: 13.1% have no history

41.6% of training rows never see a single earlier example of their own category. At 2,000 levels across 4,200 training rows, most levels appear two or three times, so for a large fraction of rows there is simply nothing legitimate to learn from. Their encoding is the global prior — a constant. Averaging permutations cannot manufacture history that does not exist.

Out-of-fold encoding wins here because it is less strict: a row in fold 3 gets the mean over folds 1, 2, 4 and 5 — around 80% of the data rather than "whatever happened to precede me". Both exclude the row's own label, which is the part that matters. Ordered TS excludes more, and at this cardinality it excludes too much.

One caveat I want to be explicit about: this is my thirty-line reimplementation of the principle, not CatBoost. Real CatBoost recomputes target statistics inside the boosting loop with a fresh permutation per tree, builds combinations of categorical features, and applies the same ordering idea to gradients. Do not read this table as "CatBoost loses to out-of-fold encoding" — read it as "the ordering principle, applied naively as a preprocessing step, runs out of history at extreme cardinality."

What To Actually Do

TARGET ENCODING: A DECISION PROCEDURE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Does the column plausibly carry signal?
  |
  +- No  -> DROP IT. Measured best in every
  |         no-signal run. Encoding a useless
  |         column only invents noise.
  |
  +- Yes / unsure
       |
       +- Fewer than ~50 levels?
       |    -> one-hot. Boring, safe, competitive.
       |
       +- Many levels, several rows each?
       |    -> out-of-fold target encoding.
       |       Best measured result here.
       |
       +- Many levels, 1-3 rows each?
            -> the column is nearly an ID.
               Expect little. Consider dropping,
               or hashing to fewer buckets.

NEVER: mean over all training rows including self.
       train 0.2466 / test 2.7975 on a column
       that contained nothing.

The tell that you have leaked is always the same shape: training loss far better than anything the problem should allow, and a validation gap that widens as you add capacity. A gap of +2.55 log loss is not overfitting. Overfitting is +0.05. That is contamination.

When Does CatBoost Help Most?

Reach for CatBoost when your table is mostly categorical with real cardinality — retail SKUs, merchant IDs, geography — and you would otherwise be hand-rolling encodings and getting them subtly wrong. Reach elsewhere when your features are numeric, where its advantage largely disappears.

Quick Reference Card

CATBOOST / TARGET ENCODING: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
THE LEAK:
  x_i = mean(y_j for j where c_j == c_i)
  includes j == i.  Row i's own label is inside
  row i's feature. One row per level => the
  encoding IS the label.

THE FIX (ordered target statistics):
  fix a permutation; for row i use only rows
  BEFORE it:
    x_i = (sum of earlier y + a*prior)
          / (count of earlier + a)
  self excluded by construction.

MEASURED (column with NO signal at all):
  naive        train 0.2466   test 2.7975
  out-of-fold  train 0.4436   test 0.5823
  drop it      train 0.4964   test 0.5464   <- best

MEASURED (column WITH signal):
  naive 2.2834 | oof 0.5624 | ordered 0.6549
  | drop 0.6352      oof wins

WHY ORDERED CAN LOSE:
  41.6% of rows had NO earlier row of their own
  level (2,000 levels / 4,200 rows). Averaging
  16 permutations did not help — the history
  isn't thin, it's absent.

THE TELL YOU LEAKED:
  train loss impossibly good + gap that GROWS
  with capacity. Overfitting is +0.05.
  Leakage was +2.55.

Key Takeaways

1. Naive target encoding leaks the label into the feature. The row's own $y$ sits in the numerator of its own $x$ .

2. It manufactures signal from nothing. On a column of random integers: train 0.2466, test 2.7975 — five times worse than deleting the column.

3. Ordered target statistics fix it by chronology, not ignorance. Each row sees only rows earlier in a random permutation; it is excluded from its own encoding by construction.

4. Out-of-fold encoding was the best real method measured — 0.5624 against 0.6352 for dropping a signal-bearing column, and it beat my ordered implementation.

5. Dropping a signal-free column beats every encoding of it (0.5570 vs 0.5863). Encoding cannot recover information that was never present.

6. Ordering can exclude too much. 41.6% of rows had no earlier example of their own level, and averaging 16 permutations did not help — the fix for absent history is not more permutations.

7. Leakage and overfitting look different. Overfitting widened the gap by 0.05. Leakage widened it by 2.55. If your training metric looks impossible, it is.

8. Check your own reimplementation before blaming the library. My thirty-line ordered TS is not CatBoost, and I said so rather than reporting a benchmark I had not actually run.

The One-Sentence Summary

Target encoding is the most widely used trick in tabular machine learning and its textbook form quietly puts each row's own answer inside that row's features — CatBoost's contribution is Ilaya's rule, that a row may be described using everything the data knew before it and nothing the data learned after, which is not a clever optimisation but a refusal to look at tomorrow's newspaper while reporting today's news.

What's Next?

1. Bagging vs Boosting — tomorrow. The settled comparison, measured rather than asserted.
2. XGBoost vs LightGBM in 10 minutes — same data, same budget, a decision procedure.
3. Hyperparameter tuning — what to search when four parameters interact.
4. Stacking — what to do when you have several good models and no idea which to trust.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

If the court at Vashti made target leakage click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever shipped a target-encoded feature without checking the fold boundaries? I have, and the model was the best I had ever built right up until the day it met data it hadn't already been told the answer to. ⚖️

The thing that unsettles me about leakage is that it never feels like cheating while you are doing it. Every step is defensible — use the data you have, summarise the category, feed the model context. The failure is assembled entirely from reasonable decisions, which is why code review does not catch it and a suspiciously good training curve does. Learn to distrust your best results specifically because they are your best results.