I built a Forge major-version predictor on a guess, and it took a review to find the answer inside my own commit

Key takeaways

• END STATE: a predictor that PARSES two manifests and reports every documented major-version trigger — not a regex over the shapes your own manifest happens to use.
• There are eleven documented triggers, on TWO pages: nine on the versions page, plus remotes and customer-managed egress — and the llm module documents its own, elsewhere.
• Our 4.0.0 had TWO independent triggers: a new scope AND the llm module. Either alone would have forced it.
• Regex extractors fail in the dangerous direction: re-indent a scopes list and every scope vanishes, so a real new scope reads as minor.
• forge version bulk-upgrade can apply eligible majors WITHOUT site-admin approval — a major is not automatically a stalled rollout.

We shipped a release of Sentinel Vault that went from 3.x to 4.0.0, and our deploy note records the reason as "major bump from the new llm module".

I decided that note was wrong. The llm module was the headline feature, the version went up, and I read that as the classic post-hoc mistake — because the same commit also added one scope, read:label:confluence, and a scope is the trigger everybody knows about. So I wrote a tool, and an article, explaining that the module was innocent and one line of YAML had done it.

That was wrong, and the correction was sitting in the same commit the whole time. The manifest comment, written when the module was added, reads:

  # Atlassian-hosted LLM (Forge LLMs) — powers Semantic AI Validations with no
  # external egress, so the app keeps its "Runs on Atlassian" badge. Adding this
  # module triggers a major version bump + admin re-consent.

Atlassian's llm module reference says the same thing outright: "Adding the *`*llm`* module to your manifest will trigger a major version upgrade."*

Two independent triggers fired on that release, not one. Either would have forced 4.0.0 by itself. The note was right, my correction was the misattribution, and the tool I built to stop people guessing was itself built on a guess — it knew about six triggers when the documentation lists eleven, across two different pages.

This is that tool, rebuilt properly, and the specific ways the first version was wrong.

Prerequisites

Node 18 or later. Verified on v24.15.0.

Two versions of a manifest.yml — most easily git show <sha>:manifest.yml for the release you are comparing against.

No Forge app, no deploy, no tunnel. The analysis is a diff of two files, so it runs offline.

Useful but optional: @forge/cli, if you want to confirm the prediction against a real deploy in step 5.

About forty minutes.

Why this matters more than a version number

A major version is not cosmetic. Atlassian's versions documentation is explicit:

By default, major version upgrades are not applied to an app installation immediately. This is because major versions involve significant changes that may require users and admins to re-consent or review the changes before continuing.

So a major stops auto-upgrading. But it is not automatically a stalled rollout, and I had that wrong too. The same page:

For eligible major version updates that don't require an escalation in privilege, you can use the forge version bulk-upgrade CLI to start, cancel, and track updates in large batches without site admin approval.

So the cost depends on whether the major escalates privilege. A new scope does; a change that does not escalate can often be pushed with bulk-upgrade, and Rolling releases let you ship code while admins approve permissions separately. The thing to avoid is a surprise privilege escalation, not a major version as such.

The rule that decides it is one sentence:

Not all permission changes trigger a major version upgrade. Only changes that require user consent, such as OAuth scopes and Atlassian app permissions, result in a major version change.

Consent, not size. That is why a workflow engine can be a minor and one line of YAML cannot.

1. Enumerate the triggers from the documentation, and note there are more than one page of them.
2. Parse the manifest instead of pattern-matching it this is the difference between a tool that works on your repo and one that works.
3. Compare the two parsed manifests trigger by trigger, respecting which removals count.
4. Prove each trigger fires, including the ones your own manifest has never used.
5. Confirm against a release that already shipped, and reconcile any disagreement.
6. Wire it into the path you deploy from.

Step 1 — Enumerate the triggers, from more than one page

The versions page lists nine bullets: adding, swapping or removing a scope; adding or swapping a content CSP option; adding or swapping an external CSP option or URL; adding a dynamic web trigger or making a static one dynamic; adding or modifying the category of an existing egress permission; flipping inScopeEUD from false to true for the first time; enabling licensing; adding or removing providers; and changing a provider client ID.

Below that list, two more in prose: "In most cases, updating your app's remote backends will result in a new major version", and adding permissions.external.configurable.enabled for Customer-managed Egress.

That is eleven. And there is a twelfth that is not on that page at all — the llm module documents its own trigger in its own reference. That is the one that caught me, and it is the reason to state the snapshot date on any list you encode:

export const TRIGGERS_SNAPSHOT = "2026-08-29";

How you know it worked: count what you encoded and compare it against the page, not against memory.

node -e "import('./predict.mjs').then(m => console.log(m.TRIGGERS.length, m.TRIGGERS.map(t=>t.key).join(', ')))"

If your count is smaller than the page's, you have already built the bug I did. Mine said six.

Step 2 — Parse the manifest, do not pattern-match it

This is the step that decides whether the tool is real.

My first version used regexes with hard-coded indentation — scopes at four spaces under permissions:. It worked perfectly on our manifest and failed silently on anything shaped differently, in the worst possible direction: the extractor returned an empty list, an empty list compared to an empty list is no change, and a genuine new scope reported minor.

A manifest is YAML. Parse it:

import { createRequire } from "node:module";
const require = createRequire(import.meta.url);
const yaml = require("js-yaml");

const get = (o, path) => path.split(".").reduce((a, k) => (a == null ? a : a[k]), o);
const asList = (v) => (Array.isArray(v) ? v : v == null ? [] : [v]);

/** Deterministic deep signature, independent of key order. */
function sig(v) {
  if (v === null || v === undefined) return "null";
  if (Array.isArray(v)) return "[" + v.map(sig).sort().join(",") + "]";
  if (typeof v === "object") {
    return "{" + Object.keys(v).sort().map((k) => `${k}:${sig(v[k])}`).join(",") + "}";
  }
  return String(v);
}

sig() is what makes reordering and reformatting safe properly. My regex version sorted raw lines to achieve the same thing, which threw away nesting — and that had a real consequence, covered in step 4.

How you know it worked: re-indent your manifest and confirm nothing changes.

python3 -c "import yaml,sys; print(yaml.safe_dump(yaml.safe_load(open('manifest.yml')), sort_keys=False))" > reindented.yml
node predict.mjs manifest.yml reindented.yml

You should see minor and triggers: none of the 9 fired. A parse-identical file that reports MAJOR means you are still comparing text somewhere.

Step 3 — Compare trigger by trigger, and get removals right

Each trigger becomes a set of comparable items, and the comparison is a set difference. The subtlety is that removal does not count everywhere.

export const TRIGGERS = [
  { key: "scopes", label: "OAuth scopes", removalIsMajor: true,
    items: (m) => asList(get(m, "permissions.scopes")).map(String) },

  { key: "external", label: "External permissions", removalIsMajor: false,
    items: (m) => Object.entries(get(m, "permissions.external") || {})
      .flatMap(([cat, v]) => typeof v === "object" && !Array.isArray(v)
        // keep the CATEGORY in the key, so moving a URL backend->client is a change
        ? Object.entries(v).flatMap(([sub, u]) => asList(u).map((x) => `${cat}.${sub}=${sig(x)}`))
        : asList(v).map((x) => `${cat}=${sig(x)}`)) },

  { key: "llm", label: "Forge LLM module", removalIsMajor: false,
    items: (m) => asList(get(m, "modules.llm")).map((x) => `llm:${x && x.key}`) },

  { key: "providers", label: "Providers", removalIsMajor: true,
    // TOP-LEVEL key — not nested under anything.
    items: (m) => Object.entries(get(m, "providers") || {})
      .flatMap(([kind, list]) => asList(list).map((p) => `${kind}:${sig(p)}`)) },

  { key: "remotes", label: "Remote backends", removalIsMajor: false,
    items: (m) => asList(get(m, "remotes")).map((r) => sig(r)) },
  // …scopes' siblings: content, webtrigger (dynamic only), licensing, configurable egress
];

Three details, each of which I got wrong first time.

Removal is a major for scopes, and not for the others. The docs list "Removing a scope" explicitly, but for content and external they list only adding and swapping — and for remotes they say the opposite outright: removing an entry, or any update that decreases a remote's scope, is a minor. A blanket "removals count" rule produces false majors.

**providers**** is top level.** Mine looked for it nested and therefore never fired at all — adding a provider, changing a client ID, swapping one provider for another, all reported minor. A trigger that cannot match is worse than an absent one, because the output looks complete.

Only dynamic web triggers count. The documented cases are adding a dynamic trigger and changing a static one to dynamic. Counting every web trigger produces false majors on ordinary module additions.

How you know it worked: run it on the real commit and read the trigger list, not just the verdict.

node predict.mjs before.yml after.yml

Real output from the commit that took Sentinel Vault to 4.0.0:

bump    : MAJOR
consent : admin approval may be required
  + OAuth scopes: read:label:confluence
  + Forge LLM module: llm:sentinel-vault-llm

Two triggers. That is the output I should have had before I wrote anything, and it is the whole reason the tool exists — not to give a verdict, but to name every reason for it, so you cannot pick the one you already believe.

Step 4 — Prove each trigger fires, especially the ones you have never used

A predictor tested only against changes your repo has actually made is untested on everything else — and "everything else" is where the surprises live, because they are the shapes you have no habits around.

Generate fixtures by mutating a parsed manifest, so they are valid YAML rather than string-spliced text:

import yaml, copy
base = yaml.safe_load(open("manifest.yml").read())
def dump(o, name): open(name, "w").write(yaml.safe_dump(o, sort_keys=False))

m = copy.deepcopy(base); m["providers"] = {"auth": [{"key": "g", "clientId": "EXAMPLE"}]}; dump(m, "x-prov.yml")
m2 = copy.deepcopy(m); m2["providers"]["auth"][0]["clientId"] = "CHANGED";              dump(m2, "x-prov-id.yml")
m = copy.deepcopy(base); m["remotes"] = [{"key": "r", "baseUrl": "https://api.example.com"}]; dump(m, "x-rem.yml")
m2 = copy.deepcopy(m); m2["remotes"][0]["baseUrl"] = "https://api.evil.com";            dump(m2, "x-rem-url.yml")

How you know it worked. Every trigger fires, and the parse-identical control does not. Real output:

  re-indent+requote (want minor)     bump    : minor
  new scope on reindented (MAJOR)    bump    : MAJOR
  providers added (MAJOR)            bump    : MAJOR
  provider clientId changed (MAJOR)  bump    : MAJOR
  remotes added (MAJOR)              bump    : MAJOR
  remote baseUrl changed (MAJOR)     bump    : MAJOR
  egress category move (MAJOR)       bump    : MAJOR
  licensing in a COMMENT (minor)     bump    : minor
  licensing enabled (MAJOR)          bump    : MAJOR

Four of those nine were broken in my first version, and every one failed in the direction that says "ship it".

providers never matched, so all three provider cases read minor. remotes did not exist as a trigger. The egress category move — taking a URL from backend to client, which the docs name explicitly — passed as minor because I sorted raw lines and threw the nesting away. And the licensing check matched the string licensing: anywhere in the file, so a comment reading # TODO: licensing: later reported MAJOR while flipping enabled: true to false reported minor.

The first two lines are the pair that matters most. A parse-identical reformat must be minor, and a real new scope on that same reformatted file must be MAJOR. My regex version got both backwards.

Step 5 — Confirm it against a release that already shipped

Now the honest test: point it at history and check it agrees with reality.

Take a release you know went minor and one you know went major, and run both. Ours:

=== the llm-module commit that shipped 4.0.0 ===
bump    : MAJOR
  + OAuth scopes: read:label:confluence

=== negative control: the workflow-engine commit ===
bump    : minor
triggers: none of the 9 fired

That second one is the whole thesis in one line. A complete workflow state engine — new logic, new UI, new stored records — routed through the existing action router on scopes the app already held. No consent surface changed, so it was a minor and it installed itself.

How you know it worked: every historical major you test must predict MAJOR, and every minor must predict minor. A disagreement is information, not a bug in the tool — go and find which trigger changed, because something did.

If you want the definitive answer for a change you are about to make, the CLI will tell you at deploy time. Atlassian's documentation describes it plainly: the linter warns before triggering the deployment, and "You can approve the deployment (effectively acknowledging the major app version bump) by running *`*forge deploy --approve MAJOR_VERSION_RULE`." That flag is the tell — if you are typing it, you are shipping a major.

A note on where the authority lives

Before building anything, be clear about which source decides.

The CLI is the authority. It runs its own linter against your manifest at deploy time, blocks when it sees a major change, and requires an explicit acknowledgement to continue. Nothing you build locally overrides that, and nothing you build locally needs to — the point of predicting is to know before you are standing in front of the block, not to argue with it.

So this predictor is a forecast, not a ruling. When the two disagree, the CLI is right and your extractor has a gap. That is a useful signal rather than an annoyance: a disagreement tells you there is a trigger you are not reading, and finding it is the whole value.

Treat the versions documentation the same way. It is the specification the CLI implements, and it moves. Anything you encode from it is a snapshot with a shelf life, which is why step 5 checks the prediction against releases that actually happened rather than trusting the model in isolation.

The triggers, and what each one actually means

Worth walking the main ones individually, because most are things people add without thinking of them as permission changes at all.

OAuth scopes. The obvious one, and the one that caught us. Adding, removing or swapping counts — swapping is the trap, because replacing read:confluence-content.all with something narrower feels like a security improvement and is still a change to what the user agreed to. Our 4.0.0 was a single added scope in a 36-line manifest diff.

External permissions. Egress. The domains your app is allowed to talk to, declared under permissions.external. Adding a backend fetch target is asking the customer to trust a new destination with their data, so of course it needs consent. This one bites teams who move a service to a new hostname and think of it as configuration.

Content and CSP options. What your app is allowed to load and render. Same logic: you are widening what can execute inside the customer's page.

Web trigger modules. A web trigger is a public URL into the customer's tenant. Adding or modifying one changes the app's attack surface in a way an admin should see, which is why it sits alongside scopes rather than with ordinary modules. This is the one that surprises people most — it feels like a module addition, and modules are usually free.

Licensing. Turning licensing on changes the commercial relationship, not just the code.

Providers. Adding or removing a provider changes who else is involved in serving the app.

Notice what is not on that list: resolvers, new UI surfaces, storage usage, functions, scheduled triggers, product event triggers, and essentially all business logic. That is the entire reason a workflow engine can ship as a minor.

Why the workflow engine was a minor, concretely

It is worth being specific about the negative case, because "we added a lot and it was a minor" sounds like luck rather than a rule.

The workflow state engine added logic, an admin surface, and new stored records. Every one of those went through machinery the app already had: an existing action router the front end already called, and storage:app, which the manifest already declared. No new scope. No new egress. No web trigger. No licensing change. No provider.

So the consent surface the customer originally agreed to did not move a millimetre, and Forge correctly treated a substantial release as a minor. It auto-upgraded, admins never saw a prompt, and nobody had to be told anything.

Put the two releases side by side and the rule stops being abstract. The release that cost every admin an approval step added one line. The release that added a subsystem cost nobody anything.

Reading the result honestly

Three failure modes to know about, because a predictor you over-trust is worse than none.

A false minor from a shape mismatch. These extractors assume a particular indentation. If your manifest nests differently — a scopes list at two spaces rather than four, an external block under a different parent — the extractor silently returns an empty list, and an empty list compared against an empty list is a minor. This is the dangerous direction, because nothing looks wrong. Step 4's fixtures exist precisely to smoke this out: if a fixture you know should be MAJOR comes back minor, the extractor is not seeing your file.

A false major from cosmetics. Reformatting, reordering or re-indenting a block can look like a change if you compare raw text. That is why the block extractors sort their lines and the scope extractor compares sets rather than sequences. If your team reformats YAML, verify that a pure reformat still predicts minor before you trust the tool on a real release.

A correct major you did not intend. The most common version of this is a scope added speculatively — declared while prototyping, never removed, never called. It costs a full major version and an approval from every admin, for a capability the code does not use. Before shipping a major, grep your source for each newly added scope's API surface and confirm something actually calls it.

What it costs you to get this wrong

The version number is not the damage. The rollout is.

A minor upgrade propagates on its own. A major sits at every installation until a site admin opens the manage-apps screen and approves it, and there is no way for you to do that on their behalf. That is fine when it is expected and planned. It is expensive when it is a surprise, for three reasons.

For the escalating kind you lose control of timing. A fix you shipped on Tuesday reaches sites on whatever day each admin happens to look, which can be never for a site with no active admin.

Your installed base fragments. Some tenants run the new version, some the old, and any support conversation now starts with working out which. If the release contained a data-shape change, you are maintaining both shapes for longer than you planned.

And you spend trust. An admin who is asked to re-consent expects the prompt to correspond to something meaningful. Asking for re-consent because of an unused scope teaches them to approve without reading, which is exactly the habit you do not want in the people guarding your customers' data.

None of that is catastrophic. All of it is avoidable by knowing, before you type forge deploy, which kind of release you have.

Step 6 — Wire it into the path you deploy from

A predictor you have to remember to run is a predictor you will not run on the release that matters.

Put it in front of the deploy so the answer arrives unasked:

git show origin/main:manifest.yml > /tmp/before.yml
node predict.mjs /tmp/before.yml manifest.yml || exit 1
forge deploy

As shipped it exits non-zero on every MAJOR, which will also block an intentional release-branch major — gate it on the branch if that matters to you.

How you know it worked: make a deliberate one-line scope change on a scratch branch and run your deploy path.

# insert INSIDE the scopes block, and pick a scope you do not already declare
sed -i '' 's/^\( *- search:confluence\)$/\1\n    - write:space:confluence/' manifest.yml
node predict.mjs /tmp/before.yml manifest.yml; echo "exit=$?"

You should see MAJOR, the scope named, and exit=1:

bump    : MAJOR
consent : admin must approve the upgrade
  + OAuth scopes: write:space:confluence
  exit=1

Then revert the line and confirm you get minor and exit=0 — a gate that cannot switch off is not a gate.

Two things I got wrong writing this test, both worth avoiding. Appending to the end of the file does not add a scope — it lands outside the scopes: block and gets attributed to whichever block happens to be last, which produced a confidently mislabelled result. And my first attempt used a scope the manifest already declared, so the predictor correctly reported minor; re-declaring an existing scope is not new consent. The tool was right and the test was wrong, which is the more common way round than people expect.

The misattribution was mine, and that is the interesting part

I want to dwell on the misattribution, because it is the reason this tutorial exists rather than a footnote to it.

Our testing document records the 4.0.0 deploy and says, in plain words, "major bump from the new llm module". I decided that was a post-hoc mistake — module is big, version went up, therefore somebody drew a lazy line. The scope was right there in the same commit, one line, unglamorous, exactly the sort of thing a person overlooks.

Both were triggers. The note was right, my correction was wrong, and the evidence was a comment in the same commit saying so — which I did not read, because I already had an explanation I liked better.

That is the failure worth naming, because it is not ignorance. I had more information than the note did — I knew about the scope — and I used the extra fact to build a more sophisticated wrong answer. A single cause is a satisfying shape, and when two triggers fire, picking the one you just discovered feels like insight.

The predictor exists to make that impossible: it does not return a cause, it returns every trigger that fired. You cannot pick your favourite from a list the tool prints in full.

The general version: when you are about to correct somebody, check whether the thing you are correcting is documented somewhere you have not looked. The llm module's trigger is on the llm module page, not the versions page. I read one page and concluded the other was folklore.

Troubleshooting

Where this fits in a release routine

The predictor answers one question, so it belongs at one point: immediately before you decide what kind of release you are cutting.

A routine that works is short. Diff the manifest against the last released version, not against your own last commit — what matters is the surface the installed base consented to, which may be several releases behind your branch. If it reports MAJOR, confirm the named trigger is intentional, and if it is a scope, confirm the code actually uses it. If it reports minor, ship and let it auto-upgrade.

The one habit worth adding is to check before writing the changelog rather than after. A major version deserves a release note that tells admins why they are being asked to re-consent, and writing that note is much easier when you know the answer is "one new scope, for the label-reading feature" rather than discovering it from a CLI error at deploy time.

That is also the sentence to put in the note. Admins approving an upgrade are being asked to trust you with something new; naming the specific thing is both more honest and considerably faster to approve than "various improvements".

What this does not tell you

It reads a manifest, so it knows about consent surface and nothing else.

It cannot tell you whether your major version is justified, only that you are about to ship one. It does not check whether your code actually uses a scope you declared — a scope you added and never called still costs a major and still asks every admin to re-consent, which is the most annoying possible reason to hold up a rollout.

And it is a mirror of one page of documentation. If Atlassian adds a seventh trigger, this predictor will confidently report minor on it. That is the standing risk with any tool built from a doc: re-read the source before a release you care about, rather than trusting a snapshot somebody took months ago.

If you only do one thing

Run the diff against the version your customers actually have installed, once, before your next release. Not against your last commit — against the released manifest.

Most teams have never done that comparison, and the answer is usually one of two things. Either nothing has moved, in which case you can ship without thinking about it again. Or something moved months ago and has been sitting on a branch waiting to surprise you, which is much better to discover on a Tuesday afternoon than during a rollout.

The whole exercise is forty minutes and it converts a recurring unknown into a check you run without thinking. That is a good trade for anything that decides whether your users get your next fix automatically or wait for an administrator to notice.

You now have a predictor that reads two manifests and answers MAJOR or minor with the reason attached, covering all six documented triggers — and, more importantly, a fixture set proving each branch fires independently plus a control that must come back negative. You confirmed it against a real major and a real minor from your own history.

The idea worth keeping past Forge: version numbers here track consent surface, not effort. A line of YAML can cost you a major version and a stalled rollout while a month of engineering ships silently. Count the manifest delta, not the diff.

More on this app's platform behaviour in the Sentinel Vault series, and the wider collection of Atlassian Forge findings.