You picked a free model because the answers looked good. Good answers are not an endpoint. An endpoint is the model plus the server plus the network. Demos pass. Pipelines stall. The model was rarely the problem.
So why do we keep benchmarking only the model? Because it is easy. You paste a prompt. You read the output. You declare a winner. The server never gets a vote.
This post is a reproducible benchmark. It measures the pair, not the model. Run it before you wire any free endpoint into CI.
The Pair, Not the Model
Most evaluations compare answers. You paste a prompt. You judge the output. You pick a winner. That measures the model. It ignores the server.
Free model access usually means a shared endpoint. A free server option means shared tenancy. Other users share the CPU, memory, and network. Your latency is their latency. Your timeout is their timeout.
Here is the scenario I keep seeing. A team evaluates a free model on Friday. The answers look great. They wire it into CI on Monday. By Wednesday, the pipeline is red. The model did not change. The server did. A neighbor started a batch job. Now every request queues behind it.
I applied the same harness to MonkeyCode's free model access and their free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I did not trust the demo. I built a harness instead.
The Harness
A benchmark needs three things. A fixed prompt set. A concurrency ladder. A pass/fail table. Here is the harness I use.
#!/usr/bin/env python3
"""Benchmark a model endpoint as a pair: model + server."""
import argparse
import asyncio
import json
import statistics
import time
import httpx
PROMPTS = [
"Say OK.",
"Classify this log line: ERROR disk full",
"Return one word: is 429 a retryable status?",
]
async def fire(client, url, payload, sem, timeout=30):
async with sem:
start = time.perf_counter()
try:
r = await client.post(url, json=payload, timeout=timeout)
return r.status_code, time.perf_counter() - start
except Exception as exc:
return type(exc).__name__, time.perf_counter() - start
async def run_level(client, url, payload, concurrency, n):
sem = asyncio.Semaphore(concurrency)
t0 = time.perf_counter()
results = await asyncio.gather(
*[fire(client, url, payload, sem) for _ in range(n)]
)
wall = time.perf_counter() - t0
ok = [lat for code, lat in results if code == 200]
errors = [r for r in results if r[0] != 200]
p95 = None
if len(ok) >= 20:
ordered = sorted(ok)
p95 = ordered[int(len(ok) * 0.95) - 1]
return {
"concurrency": concurrency,
"requests": n,
"ok": len(ok),
"errors": len(errors),
"p50": round(statistics.median(ok), 3) if ok else None,
"p95": round(p95, 3) if p95 is not None else None,
"throughput": round(len(ok) / wall, 2),
}
async def main():
ap = argparse.ArgumentParser()
ap.add_argument("--url", required=True)
ap.add_argument("--payload", default='{"prompt": "Say OK", "max_tokens": 16}')
ap.add_argument("--header", action="append", default=[])
args = ap.parse_args()
headers = {}
for h in args.header:
key, _, value = h.partition(":")
headers[key.strip()] = value.strip()
payload = json.loads(args.payload)
async with httpx.AsyncClient(headers=headers) as client:
t0 = time.perf_counter()
try:
await client.post(args.url, json=payload, timeout=60)
print(f"cold_start: {time.perf_counter() - t0:.2f}s")
except Exception as exc:
print(f"cold_start: FAILED ({exc})")
for c in (1, 2, 4, 8):
report = await run_level(client, args.url, payload, c, 20)
print(json.dumps(report))
if __name__ == "__main__":
asyncio.run(main())
The script does four things:
- Probes the cold start with one request.
- Runs 20 requests at concurrency 1, 2, 4, and 8.
- Records p50, p95, error count, and throughput.
- Prints JSON so you can store and diff reports.
Why 20 requests? Enough for a stable p95. Not enough to trigger a real rate limit. The concurrency ladder is the point. A single request hides queueing. Eight concurrent requests expose it.
Run it like this:
python3 bench_endpoint.py \
--url "https://your-endpoint.example/v1/complete" \
--payload '{"prompt": "Say OK", "max_tokens": 16}' \
--header "Authorization: Bearer $TOKEN"
The payload is yours. The endpoint schema is yours. Pass your own headers and body. The harness stays the same.
Sample output (illustrative, not from a measured run):
cold_start: 4.12s
{"concurrency": 1, "requests": 20, "ok": 20, "errors": 0, "p50": 0.84, "p95": 1.21, "throughput": 1.12}
{"concurrency": 2, "requests": 20, "ok": 20, "errors": 0, "p50": 1.10, "p95": 1.98, "throughput": 1.74}
{"concurrency": 4, "requests": 20, "ok": 17, "errors": 3, "p50": 2.31, "p95": 6.44, "throughput": 2.02}
{"concurrency": 8, "requests": 20, "ok": 9, "errors": 11, "p50": 4.87, "p95": 12.30, "throughput": 1.13}
See the pattern? Concurrency 1 looks healthy. Concurrency 4 starts dropping requests. Concurrency 8 collapses. A single prompt test would never catch this.
How to Read the Numbers
Use this table as a starting point. Adjust the thresholds to your workload.
| Metric | Green | Yellow | Red |
|---|---|---|---|
| Error rate | < 1% | 1–5% | > 5% |
| p95 latency | < 3s | 3–8s | > 8s |
| Cold start | < 5s | 5–15s | > 15s |
| Throughput at concurrency 8 | ≥ 80% of concurrency 1 | 50–80% | < 50% |
Green means the pair can handle a synchronous CI gate. Yellow means batch or async only. Red means do not wire it in.
Error rate matters more than latency. A slow answer can be retried. A dropped request cannot. Watch the error column first.
Cold start is a special case. A scheduled pipeline wakes the server. If the first call takes 20 seconds, your job times out before the model speaks. Probe it. Know the price of idle.
Where the Free Server Breaks
Free servers fail in predictable places. Here are the four I see most.
- Cold starts after idle. The first call pays the price. A scheduled pipeline wakes the server. Your p95 becomes your p100.
- Shared tenancy noise. A neighbor's batch job shifts your latency. Your numbers look different every hour. Run the ladder twice. Compare the spread.
- Rate limits without headers. 429s arrive with no Retry-After. Your retry logic guesses. Guessing makes the problem worse. Exponential backoff is the only safe move.
- Long-prompt timeouts. The model can answer. The server gives up first. Your 30-second timeout kills valid work. Measure with the real payload, not a toy prompt.
None of these show up in a prompt comparison. All of them show up in a concurrency ladder.
Wire It Into GitLab CI
A one-time benchmark is a snapshot. A scheduled benchmark is a trend. Run the harness weekly. Keep the JSON. Diff the numbers.
benchmark:
stage: test
image: python:3.12-slim
script:
- pip install httpx
- python3 bench_endpoint.py --url "$ENDPOINT_URL" --payload '{"prompt": "Say OK", "max_tokens": 16}' --header "Authorization: Bearer $TOKEN" > report.json
artifacts:
paths:
- report.json
expire_in: 30 days
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
The job needs two CI variables. ENDPOINT_URL and TOKEN. Add a schedule in GitLab. Review the artifact before you trust the trend.
Set an alert when the error rate moves from green to yellow. That is your early warning. The free server is telling you something. Listen before the pipeline breaks.
Who Should Not Use This
Skip this approach if you need sub-second p95 for user-facing features. Skip it if you need guaranteed throughput during peak hours. A free server is shared. Shared means variable. Variable means you need retries, a queue, or a paid tier.
The benchmark has limits too. It measures one endpoint, one payload, one day. It does not measure answer quality. Pair it with golden tests for that.
Also skip it if your team cannot act on the numbers. A benchmark without a decision table is just a graph. Decide the thresholds before you run. Then the output is a verdict, not a curiosity.
The Pair Is the Contract
The model answers. The server delivers. Both have to survive CI. Benchmark the pair before you trust the endpoint. Run the harness this week. The numbers will tell you more than the demo did.
