Move Your Free-Model Health Check Out of Every Pipeline

A few branches failed with the same missing label field.
I almost blamed my retry logic.
Then I looked at the raw model response.

The free route had started returning 200 OK with a shortened object.
Several jobs each spent a model call to discover the same endpoint drift.
That felt wasteful.

I wanted one shared place to learn that lesson.
Not another pipeline job that repeats it.

I use MonkeyCode's free model access and its free server option for the small shared checker below.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The problem is shared, but the quota is per branch

A GitLab CI matrix can fan out quickly.
If every job calls the same free model before doing real work, one bad endpoint turn becomes a pile of duplicate failures.

Why should five branches pay for the same bad response?
Why should five developers read the same partial JSON?

The answer is they should not.
But removing model calls from CI entirely is also wrong.
Some checks benefit from a live model.

So I moved the health check out of the pipeline.
A tiny sentinel checks the model once on a schedule.
The pipeline reads the cached result.

A sentinel, not another retry

The sentinel does three things:

• posts a canary prompt to the model route
• validates required fields in the JSON response
• exposes a fresh-enough /health endpoint for CI

If the model is healthy, CI proceeds.
If the model is broken, CI fails fast with the cached error.

Here is a minimal Python implementation I use for this pattern.
It does not need a framework.

import json
import time
import threading
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

MODEL_URL = 'https://free-model.example/v1/chat'
CANARY = {
    'prompt': 'Tag this diff: fix login timeout',
    'required': ['tag', 'confidence', 'summary'],
}
CHECK_INTERVAL = 120
MAX_AGE = 300

state = {
    'ok': False,
    'checked_at': 0,
    'latency_ms': None,
    'error': None,
    'detail': {},
}

def check_once():
    start = time.time()
    payload = {'prompt': CANARY['prompt']}
    req = urllib.request.Request(
        MODEL_URL,
        data=json.dumps(payload).encode(),
        headers={'Content-Type': 'application/json'},
        method='POST',
    )
    try:
        with urllib.request.urlopen(req, timeout=20) as response:
            data = json.loads(response.read().decode())
        missing = [key for key in CANARY['required'] if key not in data]
        if missing:
            raise ValueError('missing keys: {}'.format(missing))
        state.update({
            'ok': True,
            'checked_at': time.time(),
            'latency_ms': round((time.time() - start) * 1000),
            'error': None,
            'detail': {key: data.get(key) for key in CANARY['required']},
        })
    except Exception as exc:
        state.update({
            'ok': False,
            'checked_at': time.time(),
            'latency_ms': round((time.time() - start) * 1000),
            'error': str(exc),
            'detail': {},
        })

def loop():
    while True:
        check_once()
        time.sleep(CHECK_INTERVAL)

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != '/health':
            self.send_response(404)
            self.end_headers()
            return
        stale = time.time() - state['checked_at'] > MAX_AGE
        ok = state['ok'] and not stale
        code = 200 if ok else 503
        body = json.dumps({**state, 'stale': stale, 'ok': ok})
        self.send_response(code)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body.encode())

if __name__ == '__main__':
    threading.Thread(target=loop, daemon=True).start()
    ThreadingHTTPServer(('0.0.0.0', 8080), Handler).serve_forever()

This is not production code.
It is a small guard service.

What the sentinel caches

Each successful check stores a small JSON object:

• ok: whether the last canary call passed
• checked_at: when the check happened
• latency_ms: how long the model call took
• error: the last exception string
• detail: the subset of fields the canary requires
• stale: whether the cached answer is older than the allowed age

The stale flag matters most.
A green answer is only useful for a short window.

If the sentinel cannot refresh, it should return 503.
That prevents CI from trusting an old success forever.

GitLab CI only asks the sentinel

In CI, I do not call the model directly from every branch.
I call the sentinel first.

model_guard:
  stage: guard
  only:
    - merge_requests
  script:
    - mkdir -p sentinel
    - curl -fsS http://model-sentinel:8080/health > sentinel/health.json
    - jq -e '.ok == true' sentinel/health.json
  artifacts:
    paths:
      - sentinel/health.json
    expire_in: 10 minutes

use_model:
  stage: run
  needs:
    - job: model_guard
      artifacts: true
  script:
    - cat sentinel/health.json
    - ./run_model_job.sh

The expensive model job waits for a recent green signal.
If the sentinel says 503, the pipeline fails before spending more model calls.

The sentinel still uses the free model route.
But it uses it once on a cadence, not once per branch.

When this actually helps

The pattern catches several failure modes before CI multiplies them.

• the model returns partial JSON with missing required keys
• the route rate-limits the canary request
• latency crosses a timeout that individual jobs would hit later
• the response shape changes silently
• the endpoint floors at 200 OK while returning garbage

Each of those failures is useful to learn once.
It is not useful to relearn in every matrix cell.

Where this design fails

A single canary prompt cannot represent every possible request.
If your model drifts only on long diffs, a short canary might stay green.

The sentinel adds a moving part.
If its free server host is down, your pipeline fails even when the model is fine.

A stale cache can fool you if MAX_AGE is too high or the refresh loop dies silently.
I keep the refresh reasonably frequent and make CI reject stale responses.

Do not store model keys in the sentinel.
Load them from the hosting platform's environment variables.

Who should skip this

Skip the separate sentinel if:

• your model route is already stable and hidden behind an existing gateway
• your prompts vary so much that one canary cannot represent drift
• your environment cannot reach a shared free server
• you need exact per-request schema validation inside the pipeline anyway

This is not a replacement for local contract checking.
It is a cheap shared tripwire.

What I do before trusting the cache

I keep three rules.

First, the canary must look like the real requests.
A toy prompt produces toy confidence.

Second, the sentinel must alert outside CI.
If the health check flips red repeatedly, I want to know before the next merge request.

Third, CI must not trust stale data.
The stale flag and HTTP status exist for a reason.

That is the whole workflow.
One scheduled model call replaces many accidental ones.

Try a canary sentinel before you add another retry to your pipeline.
It is easier to build than to keep explaining the same red branch.