Leaderboards are not service tests. They show one number on one dataset. My CI does not care about that number. CI cares if an endpoint stays up, returns valid output, and fails predictably.
I applied a scorecard to MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free service means variable behavior. That is a test target, not a trust signal.
Why one-shot evals mislead
One call looks clean. It hides retries. It hides queue time. It hides JSON fields that disappear on the second call.
A benchmark table cannot answer four questions.
- Does the service recover after a bad request?
- Does p50 latency fit my CI timeout?
- Does the response shape stay stable?
- Does the endpoint fail closed or fail open?
Fail-open means invalid output can enter my pipeline. Fail-closed means a downstream job stops. I need to know which one happens before an MR touches it.
What I measure
I run a small, repeatable battery. Each metric gets a pass or fail.
- Availability: 50 calls without connection errors.
- Latency: p50 and p95 in milliseconds.
- Schema: valid JSON with the required fields.
- Determinism: same input, same expected shape.
- Failure mode: status code, body, or timeout on bad input.
- Rate-limit signal: clear 429, custom code, or silent drop.
- Recovery: the next call works after a limit.
I also send five invalid requests on purpose. A clear 400 with a JSON error body is fine. A silent drop is not. I note both.
A reproducible harness
This is an unexecuted example. Replace the URL and required fields with your own values.
import json
import os
import time
import requests
ENDPOINT = os.environ.get('MONKEYCODE_ENDPOINT', 'http://localhost:8000/v1/chat')
PROMPT = 'Return JSON with keys summary and severity.'
REQUIRED_FIELDS = {'summary', 'severity'}
def call_once(prompt):
start = time.time()
try:
r = requests.post(
ENDPOINT,
json={'prompt': prompt},
timeout=30,
)
r.raise_for_status()
body = r.json()
if not isinstance(body, dict):
return {'ok': False, 'error': 'non_object_response'}
return {
'ok': True,
'status': r.status_code,
'latency_ms': round((time.time() - start) * 1000, 1),
'valid_schema': REQUIRED_FIELDS.issubset(body.keys()),
'keys': sorted(body.keys()),
}
except requests.RequestException as exc:
status = exc.response.status_code if exc.response is not None else None
return {
'ok': False,
'status': status,
'latency_ms': None,
'valid_schema': False,
'error': type(exc).__name__,
}
except ValueError:
return {
'ok': False,
'status': None,
'latency_ms': None,
'valid_schema': False,
'error': 'json_decode_error',
}
results = [call_once(PROMPT) for _ in range(50)]
with open('results.json', 'w') as out:
json.dump(results, out, indent=2)
The script writes raw JSON to a file. I keep that file in the repo. The next person sees the same data I saw.
You can wire it into GitLab CI or any runner you use.
evaluate-free-endpoint:
image: python:3.12-slim
variables:
MONKEYCODE_ENDPOINT: $MONKEYCODE_ENDPOINT
script:
- pip install requests > /dev/null
- python evaluate_endpoint.py
artifacts:
paths:
- results.json
when: always
This job does not approve a merge by itself. It records behavior. Approval is a human decision after reading the scores.
How I read a results file
Open results.json and count the ok flags. If I see mixed statuses, I split the run. Latency spikes with a few 429s suggest a shared free pool. A flat line of timeouts suggests a network or endpoint problem.
One cold start is not a failure. Five are. Then I compare p95 against my timeout before I decide where the endpoint can run.
Thresholds, not vibes
Perfect numbers are not the goal. I set limits that match my CI.
- p50 latency under 2,000 ms.
- p95 latency under 5,000 ms.
- Schema pass rate at least 95%.
- Availability at least 99%.
- Recovery after a rate-limit passes.
If p95 goes over 5,000 ms, I stop. A retry storm can eat pipeline minutes. If schema pass rate drops below 95%, I stop. A diff labeler with missing severity is worse than no label.
One failure is not always fatal. A crash after an invalid request can be acceptable. An endless retry is not.
Where a free endpoint fits
A scorecard helps me pick the right job from the start.
| Task | Free endpoint fit |
|---|---|
| Async batch summary | Good if p95 fits the job window |
| Diff triage in MRs | Possible if schema stays stable |
| Blocking merge gate | Only with strict fail-closed checks |
| Production incident response | No, unless you own a fallback |
I do not push a free endpoint into a blocking path. I use it for triage, draft text, and batch labeling. The scorecard tells me if that changes.
What the scorecard does not tell you
It does not measure answer quality.
- It does not measure safety.
- It does not measure domain accuracy.
- It does not prove the endpoint found a real security bug.
- It does not give you an SLA.
A free tier can change tomorrow. So the scorecard is a gate, not proof. It says 'usable enough for an experiment,' not 'production ready.'
Who should not use this
Skip this if you need guaranteed uptime. Skip it if you process regulated data. Skip it if a schema change at 2 a.m. would wake you up.
A free model endpoint is a moving target. The harness tells you where the target is today.
My rule
I record the scorecard before I let the endpoint touch my repo. Three failures get fixed or removed. No new free model enters CI on a screenshot.
If you run this against a free endpoint, save the results file next to the job definition. That turns a vague promise into a reproducible check.
