Stop Trusting AI-Written GitLab CI: 12 Checks to Gate It

I stopped trusting AI-written GitLab CI files after I watched a model delete environment: production from a deploy job and still produce valid YAML. Now I make every changed pipeline pass 12 automated checks before merge, so syntax is never confused with safety.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The problem is not indentation

A YAML parser catches bad tabs. A JSON schema catches missing keys. Neither catches the failure modes that make AI-generated pipelines dangerous. Most AI-written GitLab CI mistakes are invisible to syntax tools.

I keep seeing the same mistakes:

• timeout disappears from long-running jobs.
• Runner tags are invented and match nothing in the shared cluster.
• rules conditions look logical but never evaluate to true.
• Secrets end up in variables instead of masked CI/CD variables.
• needs references a job that no longer exists.
• artifacts exposes the whole checkout or a secret directory.

Models are confident. Syntax tools are silent. Someone has to catch intent before merge. For example, deleting environment: production still leaves YAML valid, but it breaks GitLab's environment tracking and rollback.

The 12-check gate

So I built a cheap gate. It sends a changed pipeline file through a free model endpoint and demands a fixed JSON verdict. I don't ask the model to "review this pipeline." That produces paragraphs and brand names I can't verify. I ask for 12 checks with pass/fail states.

Check 12 targets a CI/CD supply-chain weakness that syntax linters rarely flag.

These checks are not a security review. They are a pre-review filter: cheap, explicit, and explainable.

Start with four checks, then add the rest

If twelve feels like too many, start with four: timeout, environment, plaintext secret, and pinned image. These cover the failures I see most often. The other eight are things I added after false merges. The goal is not to replace review. It's to make sure I read the right part of the diff when a model has already changed the wrong one.

The prompt is a schema, not a vibe

I ask the model to return JSON only. No markdown, no apologies, no follow-up questions.

You are reviewing a GitLab CI file.
Return JSON only.
Use a checks array.
Each item has id, status, and reason.
Use only pass or fail.
Do not invent a failure. If you lack evidence, pass.

The model endpoint gets the pipeline text plus the repo's allowed runner tags. That context prevents false failures on checks 7 and 8.

Then the CI job applies a jq gate:

#!/usr/bin/env bash
set -euo pipefail

schema_prompt='You are reviewing a GitLab CI file. Return JSON only. Use a checks array. Each item has id, status, and reason. Use only pass or fail. Do not invent a failure. If you lack evidence, pass.'

files="$(git diff --name-only "$CI_DEFAULT_BRANCH...$CI_COMMIT_SHA" -- '*gitlab-ci.yml' 'ci/**' || true)"
if [ -z "$files" ]; then
  echo 'No pipeline files changed.'
  exit 0
fi

for f in $files; do
  review_json="$(curl -sS "$MONKEYCODE_MODEL_URL" -H 'Content-Type: application/json' -d "$(jq -n --arg prompt "$schema_prompt" --arg content "$(cat "$f")" --arg allowed_runners "$ALLOWED_RUNNERS" '{prompt: $prompt, pipeline_file: $content, allowed_runners: $allowed_runners}')")"

  echo "$review_json" | jq -e '.checks | map(select(.status == "fail")) | length == 0' >/dev/null || {
    echo "Pipeline review failed for $f:"
    echo "$review_json" | jq '.checks[] | select(.status == "fail")'
    exit 1
  }
done

If any check returns fail, the job prints the failed reasons and blocks the merge. That fail-closed behavior is intentional: a silent false negative is worse than a loud false positive. I run this gate as a scheduled job through MonkeyCode's free server option, so I don't need to keep a laptop online.

Where this workflow falls short

I don't pretend this is a complete CI review.

• A model can mark every check pass while missing a workload-specific logic bug.
• Runner tags and include paths are hard to keep current. If the allowed list is stale, the gate either passes invalid tags or fails good jobs.
• The model may not know about GitLab features added after its training data.
• Sending pipeline content to an external endpoint is not appropriate for every repository, especially those with tightly controlled infrastructure details or embedded secrets.

Static linting still has a place. gitlab-ci-lint is deterministic and cheap. But it does not catch semantic intent. The model gate catches things a YAML parser cannot. That is the difference between checking that a file is valid and checking that a pipeline is safe.

Who should not use this

Skip it if:

• your pipelines are hand-written and already reviewed by a human who understands the whole DAG;
• your CI files contain secrets or internal topology you cannot redact;
• you need a formal, deterministic policy engine rather than a helpful second opinion.

Start with four checks, not twelve. The timeout, environment, plaintext secret, and pinned image checks catch the failures I see most often. The other eight are things I added after false merges.

Try this gate on your next pipeline merge request. Add the four checks first, then add new checks as you catch failures. If a check saves you from a bad merge, comment below or share the rule you added; I'm still deciding what check 13 should be.