I track two numbers from a third party dashboard about my extension: how many people have it
installed, and where it ranks among all extensions. They come from the same provider, on the same
page, refreshed at the same time.
One of them is stable. The other moves by about a thousand places when I read it twice.
How I found out
I started storing one row per reading rather than one row per day. That sounds like a small
bookkeeping change. It is the entire finding.
With one row per day, the second reading overwrites the first and any disagreement disappears
silently. With one row per reading, a disagreement becomes a number, and that number is the
precision of the source.
For the rank, on three separate days, the same announced date has been reported to me with two
different values: a gap of 1,063, then 1,070, then 900 places. The provider revises the rank it
attributes to a day that has already ended, and my successive readings of that same announced day do
not agree.
For the install count, over the same period, six readings of one announced day all said 39, and two
readings of the next, a day apart, both said 40. One reading of the day before the run of 39s says
- Every announced day is stable across readings. It is the sequence of days that is not monotonic, which is a different property and the one people usually mean when they say a number is reliable.
Same source. Same page. Same moments. One field is reproducible and the other is not.
What that changes in practice
It gives me a reading rule I did not have: a rank movement below a thousand places says nothing.
Not progress, not decline, not worth a sentence.
Before measuring this, I had written comments about rank movements of a few hundred places as if
they meant something. They meant nothing. They were inside the noise of a field I had never
characterised.
And it tells me which number to build on. The one that matters commercially, the install count, is
the one that happens to be reproducible. That is luck, not design, and it could easily have been the
other way round.
The uncomfortable number
Since I am recommending that people report the reliable figure, here is mine.
Forty installs, then thirty nine, then forty again.
I had been quoting this as thirty nine then forty, which reads as one user gained. Going back through
the log while checking this piece, there is an earlier reading, for an announced day two days before,
and it says forty. So across three announced days the count went down by one and back up by one. The
net movement is zero, not one.
That is what happened across a period in which I published nine pages in one day and eleven the next,
plus everything before.
I do not think that number condemns the approach yet, because most of those pages are days old and
several spent their first hours excluded from search indexes by the platforms themselves. But I am
not going to present a stable measurement as encouraging when it is not. The honest summary is: the
publication side is working, the installation side has not moved, and I know the second is measured
better than the first.
The generalisable part
I had been treating the provider as trustworthy or not, as a single thing. That is the wrong unit.
Reliability is a property of a field, not of a source. The same page can serve you one column
computed from data it holds directly, and another computed from a ranking that gets recalculated as
other people's data arrives. Nothing on the page distinguishes them. Both are rendered in the same
font.
The only way to tell them apart is to read the same value twice and see whether it agrees with
itself. That costs one extra reading and a row in a file.
So the practice I would offer: for every number you plan to make decisions with, store one row per
reading rather than one per period, and look at the disagreements before you look at the trend. The
disagreement is your error bar, and you cannot interpret a trend without it.
I would add one caution I have had to apply to myself twice. When the disagreement is small and the
trend is flattering, the temptation is to keep the trend and dismiss the disagreement as noise.
The correct order is the reverse: establish the noise first, then see what survives it.
Disclosure
I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your
Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year, and the free
tier covers 30 emails a month. The numbers above are its actual numbers, including the ones I would
rather not publish. You can find it at blueticks.io.
A source is not reliable or unreliable. Each of its columns is, separately, and only repeated
readings will tell you which is which.
