I Deleted 94% of a Neural Network's Weights and Lost 0.6 Points. Here Is the Whole Algorithm

Here is a sentence that should be strange and somehow isn't: you can throw away 90 to 95 percent of the weights in a trained neural network and get the accuracy back.

That result is not new and not fragile. LeCun published Optimal Brain Damage in 1989. Han et al. re-ran it on AlexNet and VGG in 2015. It has held up on ResNets, on BERT, on 70-billion-parameter language models. So the interesting question was never "does pruning work" — it is why a trained network is mostly empty in the first place, and what exactly you are allowed to conclude from the fact that it is.

I built the whole thing from scratch in a browser tab to find out: a 144–32–3 MLP, a hand-written backward pass, four pruning criteria, iterative pruning, and the lottery-ticket experiment with both of its controls. No libraries. Every number below came out of code you can run.

👉 Live, everything computed in your browser: https://dev48v.infy.uk/dl/day61-network-pruning.html

Why there is anything to delete

Over-parameterisation is a training aid, not a deployment requirement.

That is the whole resolution. A network with far more parameters than the task needs has a loss surface gradient descent can actually navigate — lots of redundant directions, lots of routes downhill. A network sized exactly right has a landscape full of walls. So you train big because that is what makes training work, and then you are holding a model whose extra capacity has already done its job.

You are allowed to bill for the journey and ship only the destination.

Which weights die

The question you actually want answered is: how much would the loss rise if this weight were zero? Expand the loss around the trained point. At a minimum the gradient term is roughly zero, so the leading term is quadratic — which is exactly what Optimal Brain Damage estimates with the Hessian diagonal:

score_obd = 0.5 * hessian_diag * W**2

That is expensive. Magnitude pruning replaces the entire analysis with:

score = abs(W)

A small weight multiplies its input into a small contribution, so cut the smallest. It ignores the input scale, the curvature, and every interaction between weights. And it is brutally hard to beat.

That last claim is only worth anything if you check it, which brings us to the part most write-ups skip.

The control you are not allowed to omit

Take the same sparsity. Replace the saliency score with a coin.

def mask_random(W, sparsity, rng):
    k = W.size - round(sparsity * W.size)
    idx = rng.permutation(W.size)[:k]
    m = zeros(W.size); m[idx] = 1
    return m.reshape(W.shape)

On my trained network, held-out accuracy at 80 % sparsity, no fine-tuning:

The 39-point gap between magnitude and random is the only direct evidence that |w| carries information. It is not a nice-to-have on the chart. It is the measurement. A criterion that merely tracks random has not been shown to work, however principled its derivation, and a surprising number of published ones barely clear it.

Sparsity is not speed

This is the number people quote wrong more than any other.

A 95 %-sparse weight matrix on a normal GPU runs at exactly the same speed as the dense one. The zeros are still sitting in the tensor and the kernel multiplies them with full enthusiasm. Stored densely it also takes exactly the same bytes.

To get memory you need a sparse format, and those carry index overhead — CSR stores an index per non-zero, so below roughly 70 % sparsity the compressed version can be larger. To get speed you need a kernel that skips zeros and still beats a heavily optimised dense GEMM, which usually means north of 90 % sparsity before it wins at all.

Structured pruning is the version that converts sparsity into latency: delete a whole hidden unit, channel or attention head, and the tensor genuinely shrinks.

The catch is that a neuron owns weights on both sides — the row of the incoming matrix that produces it and the column of the outgoing matrix that consumes it. Miss the second and you have left a consumer reading zeros. The test is exact and it is the best one in the whole project:

m1[dead, :] = 0;  m2[:, dead] = 0            # mask it
keep = [j for j in range(H) if j not in dead]
W1s, b1s, W2s = W1[keep], b1[keep], W2[:, keep]   # or physically delete it

assert allclose(forward_masked(x), forward_small(W1s, b1s, W2s, x), atol=1e-12)

Masking a neuron and deleting it must produce bit-identical logits. If they differ, your index arithmetic is wrong, and nothing downstream will tell you.

The half everyone forgets

The accuracy right after a big cut is the raw damage, not the result.

Cutting 90 % of my network in one shot dropped it from 93.3 % to 54.7 % — barely above the 33 % chance line. Twelve epochs of fine-tuning under the mask brought it to 92.0 %. Recovered 37.3 points. Weights that came back from the dead: 0.

That last number is the one to watch, because this is where implementations quietly break:

v = mom*v - lr_ft * (grad * mask)
W += v * mask                       # mask the UPDATE, not only the gradient

assert (W[mask == 0] == 0).all()    # exactly 0.0, not 1e-17
assert nnz_after == nnz_before

Mask only the gradient and momentum still carries a velocity from before the cut, so pruned weights drift off zero. Your 95 %-sparse network is dense again after one epoch, silently, with no error message and a suspiciously good accuracy number.

Also: use a lower learning rate than the original run. You are repairing a solution, not searching for one.

One shot is the wrong schedule

Given a target sparsity you can take it in one cut or in a sequence of smaller cuts with healing in between. The difference is bigger than the difference between most competing criteria.

mask = ones_like(W)
for r in range(1, rounds+1):
    target = 1 - (1 - p)**r                    # on SURVIVORS, not on the total
    mask   = magnitude_mask(W, target, prev=mask)
    W     *= mask
    finetune(W, mask, epochs=ft_epochs)

Each round sorts on magnitudes that already reflect the network's adaptation to the previous round, instead of on a snapshot of a model that has not yet noticed it lost anything.

Ten rounds at p = 0.25 lands on exactly 1 − 0.75¹⁰ = 94.37 % sparsity — 265 weights out of 4,704. Worth asserting that closed form, by the way: applying p to the original total instead of the survivors is a silent and very common off-by-one.

Same criterion, same data, same total compute. Only the order changed.

The price is real — IMP costs rounds × ft_epochs of training — which is why one-shot pruning still ships and why gradual magnitude pruning (prune on a schedule during the original run) is the compromise most frameworks implement.

The lottery ticket, stated precisely

Frankle and Carbin's 2019 claim is not "pruned networks work". That was thirty years old. The claim is that a dense randomly-initialised network already contains a sparse sub-network which, trained alone from the same initial values for the same number of steps, matches the full network.

The recipe: save θ₀ before training, train, prune iteratively to a mask m, then reset the survivors to m ⊙ θ₀ and retrain.

What makes it a claim rather than a demo is the control — the identical mask filled with a fresh random init. If the ticket beats that, the specific initial values carried information. If not, only the structure did. Both are interesting; they are not the same result; and running one without the other tells you nothing.

My four arms, 94.4 % sparsity, three seeds each, identical budget:

Read those honestly.

The gap to the random mask is 47 points and has never been in dispute. The structure iterative pruning found is real and it matters enormously.

The gap to the fresh init — the actual hypothesis — is 2.4 points, and it moves between seeds. That is not me failing to reproduce a result; it is what the follow-up literature found at scale. Rewinding to step 0 stops working on larger models, and the fix (Frankle et al., late rewinding) is to rewind to iteration k of the original run instead. Their Linear Mode Connectivity paper explains why: early training is dominated by SGD noise, and a mask found at the end only means something at the start once the run has become stable to that noise. Meanwhile several replications found that random pruning followed by proper retraining is a far stronger baseline than the original framing implied, and that much of the reported effect is IMP discovering good per-layer sparsity ratios rather than lucky individual weights.

Solid: IMP finds real structure, and late rewinding works. Shaky: the specific initial values are lucky.

One debugging trap worth the whole post

I wrote a finite-difference check for the backward pass and it failed. The code was correct.

numeric  = (loss(w + eps) - loss(w - eps)) / (2*eps)
rel = abs(numeric - analytic) / (abs(numeric) + abs(analytic))
assert rel < 1e-6

At convergence the true gradient has decayed towards 1e-5, while the difference of two nearly identical float64 losses bottoms out around 1e-11. The relative error explodes and you go hunting for a bug that does not exist — the check is measuring round-off, not correctness.

Gradient-check a freshly initialised network, where the gradients are large and the check has headroom. Same code, same assertion, passes at 1e-8.

What actually ships

Everything above assumed you can afford to fine-tune, which stops being true at billions of parameters. That constraint produced a family of one-shot post-training methods: SparseGPT solves layer-wise reconstruction in closed form with the inverse Hessian of the layer's own inputs, and Wanda simplifies it to a single score with no gradients at all:

score = W.abs() * X.norm(p=2, dim=0)     # |w| scaled by what it multiplies

Both reach roughly 50 % unstructured sparsity on a 70B model in GPU-hours, and both make the small-scale point again: what matters is not |w| alone but |w| scaled by the size of the thing it multiplies.

And the pattern that actually reaches production is prune → quantise → encode, with the pruning style chosen by the deployment bottleneck: unstructured if you are bound by size, structured if you are bound by latency, and 2:4 semi-structured if you are on an NVIDIA GPU from Ampere onwards, where exactly two zeros in every group of four executes natively at roughly twice the dense rate. That constraint is why 50 % sparsity is a real production number while 95 % often is not — it was chosen to match silicon, not to maximise a headline.

Report three numbers, always: non-zeros, multiply–adds, and measured milliseconds on the target device. Only the last one is not a proxy.

Verification

The page ships with the maths block extracted verbatim and run in Node against independent baselines — a brute-force top-k selection, closed-form sparsity arithmetic, a physically compacted network, and central finite differences on an untrained net. 213 assertions, all passing, plus 59 more that run in your browser while you read.

If you only take one thing from this: always plot the random control. Everything else in pruning is an argument. That is the measurement.

Live page: https://dev48v.infy.uk/dl/day61-network-pruning.html
Repo: https://github.com/dev48v/dl-from-zero