Give a Free Model a Latency Budget Before You Let It Into CI

I keep hitting the same failure pattern. A merge request passes every test. Then the pipeline stalls. Nothing in the diff changed. The free model route I call for a review is just having a slow minute.

Same prompt. Same payload. Wildly different wall-clock time.

That is not a model problem. It is an unbounded dependency problem.

CI treats unknown latency as zero latency

Most CI examples wire a model call straight into a job.

review:
  script:
    - python review_with_model.py

Then someone sets a generous timeout and walks away.

Why is that risky?

• A free route can be fast at 10:00 and slow at 16:00.
• One slow call can hold a runner for minutes.
• Retrying the job doubles the cost and still may not fix anything.

Before I let a model route into .gitlab-ci.yml, I make it earn a spot.

Measure before you integrate

I run a small probe outside the pipeline. I use MonkeyCode's free model access and free server option to call the route on a schedule without spending my CI minutes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The probe sends the same tiny payload several times and records the response time.

#!/usr/bin/env python3
import os, time, json, urllib.request

URL = os.environ['MODEL_URL']
TOKEN = os.environ.get('MODEL_TOKEN')
RUNS = int(os.environ.get('RUNS', '10'))
TIMEOUT = float(os.environ.get('TIMEOUT', '20'))

PAYLOAD = {
    'messages': [{
        'role': 'user',
        'content': 'Summarize this diff in one JSON object with keys: summary, risk, needs_attention.'
    }],
    'max_tokens': 64
}

def call_once():
    body = json.dumps(PAYLOAD).encode()
    headers = {'Content-Type': 'application/json'}
    req = urllib.request.Request(URL, data=body, headers=headers)
    if TOKEN:
        req.add_header('Authorization', 'Bearer ' + TOKEN)
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
        r.read()
    return time.perf_counter() - t0

call_once()  # warmup
times = sorted(call_once() for _ in range(RUNS))
p50 = times[len(times) // 2]
p95 = times[int(len(times) * 0.95)]
print(json.dumps({
    'runs': RUNS,
    'p50_s': round(p50, 3),
    'p95_s': round(p95, 3),
    'raw': [round(t, 3) for t in times]
}))

Run it a few times at different hours. One run is not a decision.

The latency budget table

I use three buckets.

A free model route can move between buckets in the same day. That is fine. The probe makes the movement visible.

Feed the result into GitLab CI

I keep the probe output as a JSON artifact. The CI job reads it and decides whether to call the model at all.

model_probe:
  stage: precheck
  script:
    - python ops/model_probe.py > model_probe.json
  artifacts:
    paths:
      - model_probe.json
  allow_failure: true

use_free_review:
  stage: test
  needs:
    - model_probe
  script:
    - if python -c "import json; exit(0 if json.load(open('model_probe.json'))['p95_s'] > 8 else 1)"; then echo 'skip free review'; exit 0; fi
    - python review_with_model.py

Notice what changed. The slow route can no longer stall a build silently. If p95 is over budget, the job skips the model and moves on.

Where this breaks

• Ten requests give a noisy p95. Treat it as a smoke signal, not a benchmark.
• Free routes share capacity. Measure during the hours you actually run CI.
• Cold starts can show up as latency. Check whether the first call is always the worst.
• Latency is not quality. A fast model can still produce bad JSON.

Who should skip this

Do not use this as your only gate if:

• You need a hard SLO on review time.
• Your pipeline handles regulated or high-risk changes.
• You call the model in a large concurrent fan-out.

In those cases, use a dedicated endpoint and run a proper load test. A free server is great for experiments. It is not a substitute for capacity planning.

If you already have a free server option, run the probe for a day before you add another CI dependency. The numbers are boring. That is exactly what CI needs.