The best 4-bit format in MLX is the one its own converter sets up to lose
Key takeaways
- NVFP4 is a TWO-LEVEL format. mlx-lm never sets the second level: global_scale appears zero times in mlx_lm and zero times in mlx.nn.
- That one unset parameter costs 2.31% perplexity, and turns NVFP4 from the best 4-bit result on the table into the worst — worse than a format using a quarter-bit less.
- Set it to the tensor's amax and NVFP4 improves 135 of 135 tensors, and beats 4-bit affine at the same footprint.
- I first fed it NVIDIA's published number and got 99.5% error, then wrote that up as NVIDIA and Apple disagreeing. It was a unit conversion. MLX wants that same quantity times 2688.
- At the lower footprint the ordering is unchanged: plain affine beats MXFP4 at an identical 4.253 bits per weight.
I set out to answer a narrow question — which 4-bit quantisation format keeps the most model quality at a given size — and found something more useful on the way. The best format available in MLX is also the one the standard conversion path configures to lose, and the gap is one parameter nobody sets.
I also got the diagnosis wrong first, publicly enough that the draft of this article accused NVIDIA of shipping a recipe that destroys tensors. It doesn't. That part is mine, and it is in here because the mistake is more instructive than the result.
Making the comparison fair
Most quant comparisons are not comparisons, because the formats do not use the same number of bits. A 4-bit format is never 4 bits per weight — there is per-group metadata on top, and how much depends on the group size and what the scale is made of.
MLX 0.32 supports four modes: affine, mxfp4, mxfp8 and nvfp4. Affine's scale and bias take the model's own dtype, which is bf16 here, so 16 bits each:
affine g128, 4-bit 4 + (16+16)/128 = 4.25 (16-bit scale + 16-bit bias)
mxfp4 g32, 4-bit 4 + 8/32 = 4.25 (e8m0 scale, no bias)
affine g64, 4-bit 4 + (16+16)/64 = 4.50
nvfp4 g16, 4-bit 4 + 8/16 = 4.50 (e4m3 scale, no bias)
Two exactly-matched pairings. I did not want to rest the experiment on my own arithmetic, so I summed the bytes MLX actually returns:
affine g64 parts=[uint32, bfloat16, bfloat16] total_bytes=73728 -> 4.500 bits/weight
affine g128 parts=[uint32, bfloat16, bfloat16] total_bytes=69632 -> 4.250 bits/weight
mxfp4 g32 parts=[uint32, uint8] total_bytes=69632 -> 4.250 bits/weight
nvfp4 g16 parts=[uint32, uint8] total_bytes=73728 -> 4.500 bits/weight
Byte-identical within each pair. On whole models mlx_lm.convert reports 4.253 and 4.503, the excess being 3-D convolution tensors and 1-D norms it leaves alone.
The test bed, stated precisely
Qwen/Qwen3.5-4Bat bf16, with noquantization_config. Note this is a vision-language model — in Qwen3.5 the unsuffixed name is the VLM, which inverts the Qwen3 convention. mlx-lm drops the vision tower and the MTP head at load, so what is measured is the 4.206B text model.Its attention is hybrid: 24
linear_attentionlayers and 8full_attention. Theq/k/v/orows below therefore describe 8 of 32 layers.Apple M3 Ultra, 96 GB. Reconstruction error on MLX 0.32.1; all perplexity on MLX 0.32.2 with mlx-lm 0.31.3.
Perplexity over 397,118 tokens of Python — every
.pyfile in the installed mlx-lm package, in 2048-token windows. That corpus is one library's source, substantially duplicated across near-clone model files, and almost certainly in the model's training data. It is a fair relative test because all formats see the same text, and a poor absolute one.
Reconstruction error, and the number that looked wrong
Mean relative Frobenius error against the bf16 original, over 135 projection tensors:
kind mxfp4 affine128 nvfp4 affine64
q_proj 12.204% 10.242% 11.323% 9.245%
k_proj 12.247% 10.496% 11.782% 9.438%
v_proj 12.311% 10.588% 10.868% 9.490%
o_proj 12.421% 10.748% 11.288% 9.594%
gate_proj 12.057% 10.179% 12.306% 9.210%
up_proj 11.895% 10.128% 13.039% 9.178%
down_proj 11.918% 10.312% 13.303% 9.296%
MEAN 12.047% 10.290% 12.465% 9.285%
One cell in there is impossible. NVFP4 uses more bits than MXFP4 — 4.50 against 4.25 — and reconstructs worse. A format that costs more and delivers less is usually a sign you are using it wrong, and I was.
The mistake, in full
NVFP4 is a two-level format: a per-group e4m3 scale, and a per-tensor scale underneath it so the group scales use their own range properly. MLX exposes the second one:
global_scale (array, optional): The per-input float32 scale used for "nvfp4" quantization if provided.
That is the entire documentation. It does not say how to compute the value, so I took NVIDIA's published one — amax / (448 × 6) — and tried it on a real gate_proj tensor:
no global_scale rel-err 12.297%
amax/(448*6) [NVIDIA's stored form] rel-err 99.498%
amax/448 rel-err 97.013%
amax/6 rel-err 10.531%
I wrote that up as NVIDIA's recipe destroying the tensor, and MLX's parameter meaning something different from NVIDIA's. Both wrong.
amax/(448×6) is NVIDIA's stored decode scale, and it is only meaningful inside NVIDIA's pair of equations, where a matching divide in the block scale cancels it. MLX wants the same physical quantity in a different unit — pre-multiplied by 448×6 — which is to say it wants **amax**. I handed it a number 2,688 times too small, watched the tensor collapse, and blamed the spec.
The semantics do not differ. My unit conversion did.
With the scale set correctly
global_scale = amax, per tensor:
kind nvfp4 none nvfp4 amax affine64
q_proj 11.323% 9.494% 9.245%
k_proj 11.782% 9.469% 9.438%
v_proj 10.868% 9.472% 9.490%
o_proj 11.288% 9.440% 9.594%
gate_proj 12.306% 9.502% 9.210%
up_proj 13.039% 9.500% 9.178%
down_proj 13.303% 9.482% 9.296%
MEAN 12.465% 9.488% 9.285%
tensors where global_scale=amax is NOT better: 0/135
Every tensor improves, and the spread collapses — NVFP4 went from swinging between 10.9% and 13.3% by tensor kind to sitting flat at 9.44–9.50%. That flatness is the tell that the format is now doing what it was designed to do. It beats affine on v_proj and o_proj and is within a tenth of a point elsewhere.
What it costs in model quality
Reconstruction error is not model quality, so here is perplexity. All five variants through one harness, quantising the same 128 projection tensors in place, on the same 397,118 tokens:
format bpw ppl vs bf16 weight-err
bf16 (unquantised) 16 2.1283 — —
mxfp4 4.253 2.2200 4.31% 12.047%
affine128 4.253 2.1974 3.25% 10.290%
nvfp4 (as mlx-lm ships) 4.503 2.2354 5.03% 12.465%
nvfp4 + global_scale 4.503 2.1837 2.60% 9.488%
affine64 4.503 2.1859 2.71% 9.285%
Read the middle two rows together. The same format, the same footprint, the same weights — and a 2.31% perplexity difference between the version the converter produces and the version with one parameter set.
That difference decides the whole comparison. As mlx-lm ships it, NVFP4 is the worst result on the table: worse than MXFP4, which uses a quarter-bit less. With the scale set, it is the best, edging affine at the same size.
And that scale is not reachable. global_scale appears zero times in mlx-lm and zero times in mlx.nn — I checked with control terms from the same search firing normally, and it appears 532 times in the compiled MLX extension, so the search can find it. It exists on the raw mx.quantize primitive and nowhere on the path a model actually travels. Convert a model the normal way and you get the 2.2354 row.
What survives from the smaller pairing
At 4.253 bits per weight the answer is unchanged and duller: plain affine beats MXFP4, 2.1974 against 2.2200, and on reconstruction error in every one of the seven tensor kinds. MXFP4 has no second scaling level to forget, so there is no equivalent trap and no equivalent upside.
What I would not conclude
These perplexity numbers come from simulated quantisation — round-tripping weights in place rather than converting the model — because the corrected NVFP4 cannot be produced by the converter at all. That harness quantises 128 tensors against the converter's larger set, so it scores about 1.8% better across the board and its numbers are not comparable to a converted model's. Within the harness every row is directly comparable, which is what the argument needs; across harnesses they are not.
The model is 4B and a VLM, the corpus is one Python library that is probably in training, and 24 of its 32 layers use linear attention that the weight table does not cover. The ordering is what I would carry to another model; the percentages are for this one.
I also did not measure speed. These formats exist because certain GPUs consume them natively, so on hardware with the tensor cores the calculus is different from fidelity alone. On Apple silicon the real memory ceiling decides which model fits, and format choice only decides how much quality you keep once it does.
And I am not going to reprise the tidy claim I made when I measured this at 8 bits — that weight-reconstruction error is a bad predictor of quality. That article measured layer-output error on a 30B model and perplexity on a different 4B model with a different corpus. This one measures weight error on one 4B model. Those are four differences, and a ratio between them is not a finding.
NVFP4's second scaling level is unreachable from mlx-lm, and leaving it unset costs 2.31% perplexity — enough to turn the best 4-bit format on the table into the worst, behind one that uses less space. Setting it to the tensor's amax improves all 135 tensors measured and flattens the variance across tensor kinds. At the smaller 4.253-bit footprint, plain affine still beats MXFP4.
The part I would keep if I kept one thing: a format that costs more and delivers less means you are holding it wrong. I saw that number, and my first move was to write up a defect in someone else's software. The tell was there in my own table and it took an adversarial review to make me read it properly.
