Your Karate Tests Are Green. But Is Your API Actually Covered?
A green Karate run answers one question well:
Did the tests we executed pass?
It does not answer a different question:
Did we test the API surface that matters?
That distinction becomes painful in mature API suites. You can have 300 passing scenarios, a beautiful CI report, and still have newly added operations, destructive methods, or entire resource paths that no Karate scenario exercises.
The useful coverage metric for an API suite is therefore not simply scenario pass rate. It is the relationship between the contract you expose and the tests you can map back to that contract.
This article shows a practical way to build that map using OpenAPI and an existing Karate suite, including where simple matching fails and what evidence you should retain before generating missing tests.
Pass rate and API coverage are different dimensions
Assume CI reports:
Karate scenarios: 184
Passed: 184
Failed: 0
Pass rate: 100%
That looks excellent.
Now compare the suite with the current OpenAPI contract:
GET /customers covered
POST /customers covered
GET /customers/{customerId} covered
PATCH /customers/{customerId} missing
DELETE /customers/{customerId} missing
GET /customers/{customerId}/orders missing
The suite is still 100% green. It is simply green over an incomplete slice of the API.
This is why I treat execution health and contract coverage as separate signals:
Execution health = did known tests pass?
Contract coverage = which API operations have test evidence?
Neither replaces the other.
Start with operations, not lines of code
For API testing, an immediately useful unit of coverage is an OpenAPI operation:
HTTP method + normalized path
For example:
GET /orders/{id}
POST /orders
This is more actionable than application code coverage when the question is whether your externally documented API surface is exercised.
Given this OpenAPI fragment:
paths:
/orders:
post:
operationId: createOrder
responses:
'201':
description: Created
/orders/{orderId}:
get:
operationId: getOrder
parameters:
- name: orderId
in: path
required: true
schema:
type: string
we can derive two contract operations:
POST /orders
GET /orders/{orderId}
The next job is to find evidence for those operations in Karate.
Mapping Karate scenarios to OpenAPI operations
Consider this feature:
Feature: Order lookup
Background:
* url baseUrl
Scenario: Get an existing order
Given path 'orders', orderId
When method get
Then status 200
And match response.id == orderId
A human immediately sees that it probably covers:
GET /orders/{orderId}
A coverage engine has to establish that relationship systematically.
At minimum it needs to extract:
method = GET
path = /orders/<dynamic-value>
and normalize the dynamic segment against the contract path.
Conceptually:
Karate: GET /orders/8c12...
OpenAPI: GET /orders/{orderId}
↓
match
You do not need an LLM for this basic comparison. It should be deterministic and explainable.
Normalize paths before comparing them
Literal string matching is too weak.
These can all represent the same contract operation:
Given path 'orders', orderId
Given path 'orders/' + orderId
Given url baseUrl + '/orders/' + orderId
while OpenAPI represents it as:
/orders/{orderId}
A useful mapper normalizes both sides into comparable structures.
For example:
Contract segments: [orders, {param}]
Test segments: [orders, <dynamic>]
Method: GET == GET
Then coverage can be based on structural compatibility rather than source-code spelling.
Be conservative when evidence is ambiguous. A false covered result is more dangerous than an unknown result because it hides work.
Coverage should retain evidence
A percentage alone is not enough.
Suppose a dashboard says:
API coverage: 82%
The first useful question is: why 82%?
For every covered operation, retain evidence such as:
GET /orders/{orderId}
-> src/test/java/orders/orders.feature
-> Scenario: Get an existing order
For uncovered operations, retain the opposite evidence:
DELETE /orders/{orderId}
-> no mapped Karate scenario
This turns coverage into something engineers can inspect rather than a vanity metric.
A practical result model might look like:
{
"method": "DELETE",
"path": "/orders/{orderId}",
"status": "uncovered",
"evidence": []
}
versus:
{
"method": "GET",
"path": "/orders/{orderId}",
"status": "covered",
"evidence": [
{
"feature": "orders/orders.feature",
"scenario": "Get an existing order"
}
]
}
Now the number is reproducible.
Do not confuse endpoint coverage with assertion quality
Finding a mapped scenario proves that an operation is exercised. It does not prove the scenario is strong.
This scenario technically reaches the endpoint:
Scenario: Get order
Given path 'orders', orderId
When method get
Then status 200
A more meaningful test may include contract-relevant assertions:
Scenario: Get order
Given path 'orders', orderId
When method get
Then status 200
And match response ==
"""
{
id: '#string',
status: '#string',
total: '#number',
items: '#array'
}
"""
I therefore avoid pretending operation coverage is a complete test-quality score.
Think of quality as several independent questions:
Was the operation exercised?
Did the scenario pass?
Are meaningful assertions present?
Are important response/error variants tested?
Is the test stable over repeated runs?
A useful test-management system keeps these signals separate instead of compressing all of them into one opaque score.
Positive coverage is only the first layer
OpenAPI can also expose candidate coverage dimensions inside an operation.
Suppose the contract documents:
responses:
'200':
description: Order returned
'404':
description: Order not found
'401':
description: Authentication required
A single happy-path scenario means the operation is exercised, but the documented behavior is not necessarily well covered.
You can model progressively richer coverage:
Level 1: operation coverage
GET /orders/{id}
Level 2: documented response coverage
200 covered
404 missing
401 missing
Level 3: schema / boundary coverage
required fields
enums
minimum / maximum
formats
The important part is to label these dimensions accurately. Do not report Level 1 as though it proves Level 3.
Generate from gaps, not from the whole specification
Once the comparison is trustworthy, generation becomes much safer.
Imagine the analysis produces:
12 uncovered operations
The wrong next step is:
regenerate 86 operations
The better workflow is:
OpenAPI contract
+
Existing Karate suite
↓
Operation mapping
↓
Coverage evidence
↓
Uncovered operations only
↓
Generate candidate tests
↓
Review + execute
This matters because mature suites contain engineering intent that does not exist in OpenAPI: authentication helpers, setup flows, domain fixtures, custom assertions, tags and team conventions.
Targeted generation reduces the chance of replacing that intent with generic generated code.
Example: turning one coverage gap into a test
Suppose coverage identifies:
DELETE /orders/{orderId} — uncovered
The contract says:
/orders/{orderId}:
delete:
operationId: deleteOrder
parameters:
- name: orderId
in: path
required: true
schema:
type: string
responses:
'204':
description: Deleted
'404':
description: Not found
Do not invent a random order ID and call that complete.
First inspect how the repository creates valid orders. If an existing reusable feature exists:
* def created = call read('classpath:helpers/create-order.feature')
* def orderId = created.response.id
then a generated candidate can fit the suite:
Scenario: Delete an existing order
* def created = call read('classpath:helpers/create-order.feature')
* def orderId = created.response.id
Given path 'orders', orderId
When method delete
Then status 204
A second candidate can address documented negative behavior:
Scenario: Delete an unknown order
Given path 'orders', 'does-not-exist'
When method delete
Then status 404
That is far more useful than blindly producing one isolated scenario per OpenAPI operation.
What happens when the specification changes?
Coverage becomes even more valuable when it is recomputed after contract changes.
Suppose yesterday's contract contained:
GET /orders/{id}
POST /orders
and today's contains:
GET /orders/{id}
POST /orders
PATCH /orders/{id}
The interesting result is not simply "OpenAPI changed."
It is:
New operation:
PATCH /orders/{id}
Mapped scenarios:
none
Coverage state:
uncovered
That is actionable change impact.
Likewise, if a response schema adds a required field, the tool should distinguish that from a completely new operation. The likely test impact is different.
This is where contract coverage starts becoming a maintenance mechanism rather than a reporting feature.
Where AI helps — and where it should not decide the truth
AI is useful after deterministic evidence exists.
Given:
PATCH /orders/{id} is uncovered
plus the operation schema and nearby repository patterns, an AI assistant can help propose:
- realistic domain scenarios
- useful boundaries
- repository-consistent naming
- stronger assertions
- reuse of nearby setup helpers
But I would not ask the model to be the source of truth for whether the operation is covered.
The safer split is:
Deterministic layer
parse OpenAPI
index Karate
map operations
identify gaps
retain evidence
AI-assisted layer
interpret domain intent
suggest scenarios
adapt to local style
propose repairs
That separation makes the result inspectable when the model is wrong or unavailable.
How Karate Test Management approaches this
This workflow is now part of Karate Test Management for VS Code.
The current extension can analyze an OpenAPI specification by itself, or combine the specification with Karate feature files for exact scenario-to-endpoint mapping. Coverage gaps enter the Quality workflow, and missing-test generation can start from those gaps rather than regenerating the entire API suite.
The same workspace also keeps execution, run history, flakiness, specification-change findings and repair review alongside coverage. That matters because an uncovered endpoint, a failing endpoint and a flaky endpoint are three different engineering problems.
The implementation deliberately keeps deterministic coverage available without an AI provider. AI enhancement is optional and uses the evidence produced by the underlying workflow.
A practical coverage review for your own Karate repository
Even without tooling, you can apply the same model manually or in a script:
- Parse the current OpenAPI document into
(method, normalized path)operations. - Index Karate scenarios and extract the HTTP method and resolved/structural path evidence you can establish.
- Map tests to operations conservatively.
- Mark ambiguous mappings as unknown instead of covered.
- Keep the feature and scenario name as evidence for every match.
- Review uncovered operations by risk, not alphabetically.
- Generate or write tests only for justified gaps.
- Re-run the mapping whenever the contract changes.
The key output should look less like a single percentage and more like an engineering queue:
HIGH DELETE /customers/{id} uncovered
HIGH POST /payments/refunds uncovered
MEDIUM PATCH /orders/{id} uncovered
LOW GET /catalog/metadata uncovered
Now API coverage can influence release decisions.
Green is necessary, not sufficient
A passing suite is good evidence about the tests you ran.
It is not evidence about tests that do not exist.
For contract-driven APIs, comparing Karate scenarios with OpenAPI gives you a practical way to expose that blind spot. Keep the mapping deterministic, retain evidence, distinguish endpoint coverage from assertion quality, and generate from gaps instead of repeatedly generating the whole suite.
That is the difference between asking "are my tests green?" and asking the more useful question:
What part of my API do those green tests actually protect?
Karate Test Management is open source if you want to try the workflow:
