Locking in Forge KVS: FAIL_IF_EXISTS works, TTL leases do not

Key takeaways

• FAIL_IF_EXISTS exists from @forge/kvs 1.3.0 (2026-02-03) and is atomic — one winner in twelve concurrent writes, eleven KEY_CONFLICT at HTTP 409.
• Adding ttl to that call does not make a lease. An expired, unswept key still blocks the write — measured here at 8m35s past a 30-second lease, with the sweep landing between 8m35s and 18m37s.
• KEY_CONFLICT and CONDITIONAL_CHECK_FAILED appear on neither storage error-handling page, both of which claim to list all possible codes.
• e.name is 'ForgeKvsError' even on a ForgeKvsAPIError. Branch on instanceof or e.code.
• Real mutual exclusion is a custom entity plus a conditional transact().set() — also one winner in twelve, at four to five times the latency.
• A custom entity integer is int32, so Date.now() will not fit. Use float. Inside a transaction the type error degrades to a bare 422 with no property name.

In December 2025 a Marketplace partner posted a question on the developer community that has a very short and very wrong answer. Chris at DigitalRose was migrating a Connect app to Forge Runs on Atlassian, and needed the thing every Connect app has and no Forge app obviously does:

If multiple users, functions, threads happen to run concurrently it is critical that the (load, update, and save) is locked to avoid corrupting or overwriting data. In Connect we do this with a simple redis mutex lock. What is the equivalent inside Forge?

The first reply said Forge has no equivalent. Specifically, that Forge "doesn't provide a Redis-style distributed mutex or atomic operations (such as CAS) in the standard Storage API, so implementing reliable locking on top of @forge/kvs is very hard in practice." The recommendation was to move the data into @forge/sql and use UPDATE … WHERE version = :currentVersion.

That was a reasonable read of the platform at the time. It is not true now, and the gap between those two statements is where apps get built wrong. This walks the whole thing: the primitive that does exist, the one-line change that quietly destroys it, and the pattern that survives twelve concurrent invocations. Every number below came off an app I deployed to a test site for this, not off a docs page.

The primitive that exists

@forge/kvs has an atomic create-if-absent. It is the keyPolicy option on kvs.set:

import { kvs } from '@forge/kvs';

await kvs.set('lock:issue-10021', { holder: 'inv-abc' }, { keyPolicy: 'FAIL_IF_EXISTS' });

If the key exists, the write throws instead of overwriting. That is lock acquisition.

The reason nobody found it is timing plus documentation. FAIL_IF_EXISTS does not exist in @forge/kvs 1.2.4, published 2025-12-15, and first appears in 1.3.0 on 2026-02-03. I bisected the published tarballs to check rather than trusting a changelog, because @forge/kvs still ships no CHANGELOG.md at all — verified again on 2.0.4:

npm pack @forge/kvs@1.2.4 @forge/kvs@1.3.0 @forge/kvs@2.0.4
for v in 1.2.4 1.3.0 2.0.4; do
  mkdir -p x$v && tar xzf forge-kvs-$v.tgz -C x$v
  echo "$v: $(grep -rl FAIL_IF_EXISTS x$v/package/out/interfaces | wc -l) interface files"
done

That prints 0 for 1.2.4 and 1 for the other two. So the December thread was correct on the day it was written and went stale six weeks later, which is the worst possible failure mode for an accepted answer.

The values are documented now, on the KVS API reference, under "Change write conflict strategy". They are still absent from the older storage-api-basic page, which mentions keyPolicy once and never says what you may pass it. I fetched both pages and counted: FAIL_IF_EXISTS is 4 hits on the first and 0 on the second, with keyPolicy present on both, so that is a real silence and not a failed fetch.

Does it actually hold?

An atomic write is a claim about behaviour under contention, and the only way to know is to contend. I built a small Forge app with a web trigger per experiment and installed it on a test site. The whole acquire handler is this:

// Web trigger. queryParameters values arrive as ARRAYS, so index them.
const json = (o) => ({ statusCode: 200,
  headers: { 'Content-Type': ['application/json'] }, body: JSON.stringify(o) });

export async function acquire(req) {
  const key = req.queryParameters.key[0];
  const tag = req.queryParameters.tag[0];
  try {
    await kvs.set(key, { holder: tag, at: new Date().toISOString() },
      { keyPolicy: 'FAIL_IF_EXISTS' });
    return json({ acquired: true, tag });
  } catch (e) {
    return json({ acquired: false, tag, code: e.code, status: e.responseDetails?.status });
  }
}

Twelve concurrent curls at one fresh key:

for i in $(seq 1 12); do
  ( curl -s "$TRIGGER?key=race-1&tag=w$i" > /tmp/race/$i.json ) &
done; wait

One winner. Eleven losers, all with the same code. Latency across all twelve ran 37–93 ms, median 52.

So the primitive is real and it holds. If your app needs "only one invocation may do this", and you are happy releasing the lock yourself, you can stop reading here and go use it.

The one-line change that breaks it

Nobody stops there, because a lock you must release yourself is a lock that leaks the moment a function throws, times out, or hits a rate limit mid-hold. The obvious fix is sitting right in the same options object:

await kvs.set(claimKey, { holder: tag },
  { keyPolicy: 'FAIL_IF_EXISTS', ttl: { value: 30, unit: 'SECONDS' } });

Create-if-absent, plus an expiry, equals a self-releasing lease. The types allow it — PolicySetOptions extends SetOptions, and SetOptions is where ttl lives — so it compiles, deploys and works in testing. It is also wrong.

The KVS docs describe TTL expiry as asynchronous, and the sentence is worth reading closely:

Expired data is not removed immediately upon expiry. Deletion may take up to 48 hours. During this window, read operations may still return expired results. If your app requires strict expiry semantics, request EXPIRE_TIME metadata and ignore values where expireTime is in the past.

Every word of that is about reads. There is no equivalent statement about writes, and a lease does not care about reads — it cares whether the next FAIL_IF_EXISTS succeeds. If a ghost key still counts as existing, a 30-second lease can hold for up to 48 hours.

I could not settle that from the documents, and a sweep that happens to fire on a dev site is an observation, not a contract. So I asked the question on the developer community on 2026-08-12. JerryZhao, from the Forge Storage team, answered on 2026-08-17:

Yes, an expired but not-yet-swept key is still visible to FAIL_IF_EXISTS, so the write will fail until the key is deleted. TTL should therefore not be relied on for precise lease semantics.

That is the whole trap in one sentence, on the record, from the team that owns the storage layer.

Reproducing the ghost

A staff answer is the strongest evidence available for an unobservable platform contract, but it costs nothing to watch it happen. I wrote a key with a 30-second TTL and then kept trying to take it:

05:09:26Z  write   ttl 30 SECONDS          → acquired
05:09:28Z  read    expireTime 05:09:56Z    → value present
--- expiry ---
05:10:23Z  write   FAIL_IF_EXISTS  (+27s)  → KEY_CONFLICT 409   read → value still present
05:11:25Z  write   FAIL_IF_EXISTS  (+89s)  → KEY_CONFLICT 409   read → value still present
05:13:28Z  write   FAIL_IF_EXISTS  (+212s) → KEY_CONFLICT 409   read → value still present
05:18:31Z  write   FAIL_IF_EXISTS  (+515s) → KEY_CONFLICT 409   read → value still present
05:28:33Z  write   FAIL_IF_EXISTS (+1117s) → ACQUIRED           the ghost was finally swept

The lock held for at least eight minutes thirty-five seconds past a thirty-second lease, and the read returned the value every single time. The prober was still running when this article went up, and it caught the ending: on the next probe the write succeeded, so the sweep landed somewhere between 8m35s and 18m37s after expiry — nowhere near the documented 48-hour ceiling. Do not build on that. It is one observation, on one key, on one dev site, and the number a lease has to survive is the worst case, not the case I happened to draw. The docs say up to 48 hours and the Storage team says TTL is not lease semantics; a fifteen-minute sweep does not soften either statement, it just means my own data cannot tell you where the ceiling really is.

If you have already shipped FAIL_IF_EXISTS plus ttl as a lease, it is not doing what the code reads like. It is doing exactly what it says on line one — refusing to overwrite an existing key — for an unbounded time after your lease expired.

What the failure actually looks like

Before the fix, the error object, because you cannot branch on something you cannot identify. Here is what a FAIL_IF_EXISTS conflict really carries, dumped off a live invocation:

{
  name: 'ForgeKvsError',
  message: 'Provided key already exists and cannot be overwritten',
  code: 'KEY_CONFLICT',
  packageVersion: '2.0.4',
  responseDetails: {
    status: 409, statusText: 'Conflict',
    httpMethod: 'POST', httpPath: '/api/v1/set', responseBodyLength: 89
  }
}

Two things there will bite you.

The constructor is ForgeKvsAPIError, but name says ForgeKvsError. That is not a mistake in my dump — the parent class sets this.name in its constructor and the subclass never overrides it, so e.name === 'ForgeKvsError' for both. Discriminating by name silently lumps API errors in with everything else. Use instanceof or the code field:

import { ForgeKvsAPIError } from '@forge/kvs';

const isConflict = (e) => e instanceof ForgeKvsAPIError && e.code === 'KEY_CONFLICT';

The second is that KEY_CONFLICT is not documented. The KVS error handling page opens with "The following tables lists all possible error codes", and then lists eleven: four key-related, two value-related, five query-related. KEY_CONFLICT is in none of them, and neither is CONDITIONAL_CHECK_FAILED from the next section. The custom entity equivalent renders sixteen codes and is missing the same two. Both pages carry a commented-out block in their own source reading "commenting this out for now until we can figure out how to document these codes", so the incompleteness is known. Take these two from this article, not from the enumeration.

The pattern that holds

For strict mutual exclusion you need compare-and-set, and on Forge that means a custom entity plus a conditional transaction. This is the route the December thread eventually reached — AaronMorris1 corrected the no-CAS claim in post 8, and the original responder conceded it in post 9 — and it is the route the Forge Storage team named as the current recommendation.

The restriction that is easy to miss: conditions are custom-entity-only. In the shipped @forge/kvs 2.0.4 types, TransactionBuilder.set takes its conditions inside EntityConditions<T> = { entityName: string; conditions?: BaseFilter<T> }, where entityName is required. There is no way to express a condition without naming an entity, so a plain KVS key can join a transaction for atomicity but can never carry a condition. The wire layer one directory away types entityName as optional, so the transaction protocol itself would accept a conditioned plain key — the public builder just gives you no way to ask for it.

Declare the entity in the manifest:

app:
  runtime:
    name: nodejs22.x
  storage:
    entities:
      - name: lease
        attributes:
          holder:
            type: string
          version:
            type: integer
          expiresAtMs:
            type: float
          expiresAtIso:
            type: string
        indexes:
          - version
permissions:
  scopes:
    - storage:app

Then acquire by reading the current record and writing a new one guarded by the version you just read:

import { kvs, Filter, FilterConditions } from '@forge/kvs';

const LEASE = 'lease';

export async function acquireLease(ekey, holder, holdMs = 30000) {
  const now = Date.now();
  const current = await kvs.entity(LEASE).get(ekey);

  // Free if there is no record, or the holder's own clock has run out. The
  // expiry is ours to enforce, not the platform's — see the TTL section.
  if (current && current.expiresAtMs > now) {
    return { won: false, reason: 'held', by: current.holder };
  }

  const next = {
    holder,
    version: (current ? current.version : 0) + 1,
    expiresAtMs: now + holdMs,
    expiresAtIso: new Date(now + holdMs).toISOString(),
  };

  // Guard on the version we read. If another invocation took it between our
  // read and this write, the version no longer matches and the whole
  // transaction aborts.
  const guard = current
    ? new Filter().and('version', FilterConditions.equalTo(current.version))
    : new Filter().and('version', FilterConditions.notExists());

  try {
    await kvs.transact().set(ekey, next, { entityName: LEASE, conditions: guard }).execute();
    return { won: true, version: next.version };
  } catch (e) {
    if (e.code === 'CONDITIONAL_CHECK_FAILED') return { won: false, reason: 'race' };
    throw e;
  }
}

Firing twelve of those at once taught me something I did not expect. Eleven of them never reached the transaction at all — they lost at the read, coming back held because the winner's write had already landed by the time they looked. That is the happy path doing its job, and it is also why a natural concurrency test can pass without ever exercising the condition you are relying on.

So I tested the condition directly instead: seed a record at version 0, then fire twelve transactions that all condition on version 0, with no read in front to serialise them.

One winner, eleven clean losses, 212–319 ms with a median of 239. The condition is genuinely atomic. Against the earlier table — 37–93 ms, median 52 — the conditional transaction is roughly four to five times the round trip, and the read in front of it in real use makes it two trips instead of one. That is the price of correctness here, and it is worth knowing before you put it on a hot path. If you are batching, note that a transaction caps at 25 operations, each key may appear only once in it, and the whole payload is limited to 4 MB.

Two details about the condition itself, both measured rather than inferred. The attribute you condition on does not have to be in indexes — I conditioned on holder, which is declared but not indexed, and got a clean success on a match and CONDITIONAL_CHECK_FAILED on a mismatch. Index version because you may want to query it, not because the condition needs it.

The one that will actually cost you: the condition's value must match the attribute's declared type, and nothing in TypeScript will tell you otherwise. FilterConditions.equalTo is typed (value: string | number | boolean), so comparing an integer attribute against the string "0" compiles perfectly and fails at runtime — as a bare 422, with no hint that a type is involved:

That third row is easy to hit, because a version arriving from a query string, a resolver payload or JSON.parse of anything is a string until you make it a number.

Storing the expiry, and the 422 that hides why

This one cost me twenty minutes, because the natural thing to store in a lease record is the moment it expires, and the natural type for that is wrong.

type: integer on a custom entity attribute is a signed 32-bit integer. The manifest reference does document this — minimum -2,147,483,648, maximum 2,147,483,647 inclusive — and I probed the boundary to be sure it is enforced exactly as written:

Date.now() is about 1.787 × 10¹², which overruns that ceiling by a factor of roughly 832. A millisecond epoch has not fitted in an int32 since 1970. Store it as float, which the same page gives 38 digits of base-10 precision — I wrote 1787116436381 and read back 1787116436381, exact. Seconds-since-epoch in an integer also works, right up until 19 January 2038, which is a fine trade if you like that sort of thing.

The second trap is what happens when you get the type wrong, and it depends on how you write. A direct kvs.entity(...).set() with an out-of-range integer tells you precisely what is wrong:

400  INCORRECT_PROPERTY_TYPE
     Data type for property "expiresAt" is defined as "integer"

The identical bad value inside kvs.transact() gives you this instead:

422  UNPROCESSABLE_ENTITY
     Request cannot be processed due to one or more semantic errors

No property name, no type, nothing to grep for. If a transaction returns 422 and you cannot see why, pull each operation out and run it as a direct entity write to find out which property the platform is objecting to. I would not have found it any other way.

What this costs you

An app-level lock never locks the Jira issue or the content property, only your own record. If something else writes that issue while you hold your lease, your lease did nothing. This is coordination between your own invocations and nothing more.

Lock patterns invite polling, and polling is expensive here. KVS is capped per installation at 1000 requests per second with 4000 reads and 4000 writes per minute, where request size is counted in 10 KB units, so a 25 KB write costs three against the 4000. A spin-wait across concurrent users burns the budget your app needs for real work, and it burns it against the same points-based limits everything else competes for. Prefer a short lease and retry with backoff.

And you cannot inspect the conflict cheaply. returnMetadataFields — the only way to get EXPIRE_TIME back from a write — appears on exactly one options variant in the shipped types, OverrideAndReturnSetOptions, which pins keyPolicy to the literal 'OVERRIDE'. The overload that accepts keyPolicy at all returns Promise<void>. So after a conflict you must re-read the key, which is a second round trip into the window you were trying to close. The docs' own strict-expiry mitigation is structurally unreachable from the write that needs it.

If you are wiring this into a workflow validator or post-function, the invocation you are trying to deduplicate may arrive twice for reasons that have nothing to do with your code. We hit exactly this in CogniRunner, our AI workflow-rule app for Jira — full disclosure, it is one of ours. The claim function keys on transition.executionId where the payload carries one, precisely so the key is never reused and a ghost does no harm, and falls back to comparing a stored claimedAt at conflict time rather than trusting the TTL. That fallback is the best-effort pattern the Storage team described, and it is the part we are moving to a custom entity. The validator and condition split decides how often you get called in the first place, which is the cheaper lever.

Where this leaves the December thread

The accepted answer on that thread is the custom entity one, and it is correct. So, for what it is worth, is the @forge/sql route it corrected. Between them they still leave a reader on 2026 packages without the plain-key primitive, because it did not exist when either was written. The current shape is:

1. Only one invocation may act kvs.set with keyPolicy: 'FAIL_IF_EXISTS', and release it yourself. One round trip, 37–93 ms measured, and it holds under concurrency.
2. You need the lock to release itself do not reach for ttl. Store your own expiry and enforce it in your code, because the platform's expiry is not visible to the write you care about.
3. You need strict mutual exclusion custom entity with a version attribute, kvs.transact().set() with a Filter condition on that version, and branch on CONDITIONAL_CHECK_FAILED.
4. You are storing a timestamp float, never integer, unless you are happy through 2038.

The open piece is conditional writes on plain KVS keys. The Storage team said they are "looking into additional conditional-write support for KVS" and would review the documentation. I checked before publishing: @forge/kvs 2.0.5-next.0, the current next tag, has an out/interfaces directory that is byte-identical to 2.0.4's, so nothing has landed on that channel yet. And the TTL section of the KVS API page still carries only the read-side mitigation, unchanged as of today. When that changes, step 3 gets a lot shorter.

So the open question narrows rather than closes: I have one sweep at roughly fifteen minutes and the documentation says up to forty-eight hours. If you have watched a ghost key survive materially longer than mine did, that number is worth more than either of ours, because it is the tail that decides whether this is a workaround or a stuck app.