I Automated My Entire GitOps Security Stack. The First Thing It Blocked Was My Own Salary.

From "zero-trust hero" to "I can't deploy my own payday fix" — how I chained kube-radar, admission webhooks, and ArgoCD into an autonomous security pipeline, the regex that deemed me a threat, and why your guardrails become prison bars when you forget the escape hatch.

The Setup

It started with a single YAML file that should never have made it to production.

I was reviewing a pull request for VehicleMetrics at 10 PM on a Thursday. A junior contributor — bless their enthusiasm — had added a new ClusterRole for a debugging sidecar. It looked innocent enough:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vehiclemetrics-debug
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

Cluster-admin in a trench coat. One kubectl apply away from "we don't know who owns this cluster anymore."

I caught it because I was looking. But what about the PRs I don't review? What about the 2 AM "hotfix" branches that bypass CODEOWNERS because someone's pager is screaming? What about the Helm chart I copy-pasted from Stack Overflow that binds a service account to pods/exec — the permission that lets anyone kubectl exec into a running pod and dump environment variables?

I manage Kubernetes at Siemens professionally. I run bare-metal clusters in my closet obsessively. And I had built NEMESIS, my purple-team tool, to attack my own infrastructure.

But I had never built something to stop the attack before it started.

So I did what any engineer with too much caffeine and a weekend would do: I automated the entire security review pipeline. No human in the loop. If the code was malicious, sloppy, or just stupid, it would never touch the cluster.

The system worked perfectly.

Then Friday at 4:47 PM happened.

Why Not Just Use Branch Protection?

You're thinking: this is a git problem, not a Kubernetes problem.

Branch protection, CODEOWNERS, required reviews — I had all of it. Here's why it's not enough:

1. YAML is a liar

That wildcard ClusterRole? It passed yamllint. It passed helm lint. It passed a human reviewer who was looking at 14 files in a 3,000-line PR. The danger wasn't in the syntax. It was in the semantics.

2. Security is boring until it's catastrophic

Nobody wants to be the reviewer who blocks a PR for three hours debating whether pods/exec is necessary. So they approve it. I know because I've done it.

3. "LGTM" is not a security control

A thumbs-up emoji doesn't enforce least privilege. A required reviewer count doesn't understand RBAC. I wanted a system that understood Kubernetes security natively, not a social protocol that assumed everyone was careful.

The Stack I Built

I took three tools I already trusted and wired them into ArgoCD's deployment pipeline:

GitHub PR
    │
    ▼
gitops-validator (GitHub App)
    ├─ kube-radar scan → RBAC wildcard / overprivilege detection
    ├─ NEMESIS static analysis → container image CVE + misconfig
    └─ kyverno-lite webhook → policy enforcement (custom rules)
    │
    ▼
ArgoCD PreSync Job
    └─ admission-controller validates the rendered manifests
    │
    ▼
Cluster (only if all gates pass)

Tool 1: kube-radar (my own Go CLI)

I wrote this when I was learning Go. It parses Kubernetes RBAC resources and scores them by risk. Wildcards = instant block. pods/exec, secrets/*, clusterroles/* without namespace restriction = flag for human review.

I containerized it and turned it into a GitHub Actions job.

# .github/workflows/gitops-security.yml
jobs:
  rbac-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: kube-radar scan
        uses: docker://ghcr.io/le-beltagy/kube-radar:v0.3
        with:
          args: scan --path ./manifests --severity critical --fail-on-critical

Tool 2: NEMESIS static analysis

The same purple-team engine that lives in my cluster as a DaemonSet has a scan mode. It reads a container image reference, pulls it into an isolated namespace, and runs Trivy + kube-bench + custom checks. If the image contains a known CVE with CVSS > 7.0, the pipeline fails.

Tool 3: kyverno-lite (custom admission webhook)

I didn't need all of Kyverno's DSL. I needed four hard rules:

1. No container runs as root
2. No image uses latest tag
3. No RBAC rule has * on apiGroups, resources, AND verbs
4. Every deployment must have resources.requests set

So I wrote a lightweight admission webhook in Go — just 400 lines — using controller-runtime. It receives AdmissionReview requests from the Kubernetes API server and returns allowed: true/false.

// webhook.go — the RBAC gatekeeper
func validateRBAC(req *admissionv1.AdmissionRequest) bool {
    var role rbacv1.ClusterRole
    json.Unmarshal(req.Object.Raw, &role)

    for _, rule := range role.Rules {
        // Rule 3: The "Deadly Asterisk"
        if slices.Contains(rule.APIGroups, "*") &&
           slices.Contains(rule.Resources, "*") &&
           slices.Contains(rule.Verbs, "*") {
            return false // ❌ DENIED
        }
    }
    return true // ✅ ALLOWED
}

I packaged it as a ValidatingWebhookConfiguration:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: gitops-security-webhook
webhooks:
  - name: rbac.gitops.lebeltagy.io
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: ["rbac.authorization.k8s.io"]
        apiVersions: ["v1"]
        resources: ["clusterroles", "roles"]
    clientConfig:
      service:
        name: gitops-webhook
        namespace: security
        path: "/validate-rbac"
    failurePolicy: Fail
    admissionReviewVersions: ["v1"]
    sideEffects: None

failurePolicy: Fail. This is the critical line. If the webhook is down, nothing gets deployed. I wanted security over availability. I would regret this later.

The ArgoCD Integration

ArgoCD has a feature most people ignore: PreSync hooks. You can run a Kubernetes Job before any sync operation. If the Job fails, the sync aborts.

I created a PreSync Job that:

1. Renders the Helm chart
2. Runs kube-radar against the rendered manifests
3. Runs NEMESIS against the container images referenced in the manifests
4. Sends a Slack notification with the scan results

apiVersion: batch/v1
kind: Job
metadata:
  name: gitops-security-gate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: validator
          image: ghcr.io/le-beltagy/gitops-validator:v1.2
          env:
            - name: REPO_URL
              value: "https://github.com/le-beltagy/vehiclemetrics"
            - name: TARGET_REVISION
              value: "HEAD"
      restartPolicy: Never

If this Job exits with code 0, ArgoCD deploys. If it exits with code 1, the sync is blocked and the Application shows SyncFailed.

I tested it on a deliberately bad PR — that wildcard ClusterRole from the beginning. The pipeline caught it. The PR was blocked. The cluster stayed safe.

I felt like a god.

Friday, 4:47 PM

It was the last workday of the month. Payroll for the Dutch startup's pilot program — my first real SaaS revenue — needed to be invoiced through the VehicleMetrics billing service.

The billing service had a bug. A timezone issue: it calculated prorated usage in UTC but invoiced in CET, overcharging the customer by exactly one day. The customer noticed. I needed to ship a fix before 5 PM or the invoice would go out wrong.

I wrote the fix in 12 minutes. One line changed in a Python utility. Tests passed. I pushed, merged, and watched ArgoCD.

The PreSync Job started.

It failed.

[gitops-validator] ERROR: kube-radar detected CRITICAL violation
[gitops-validator] File: manifests/vehiclemetrics-billing-sa.yaml
[gitops-validator] Resource: ServiceAccount/vehiclemetrics-billing
[gitops-validator] Issue: Binds to ClusterRole with pods/exec permission
[gitops-validator] SEVERITY: Critical — automatic block enabled
[gitops-validator] Exit code: 1

Wait. What?

I hadn't changed anything in vehiclemetrics-billing-sa.yaml. That file had been in the repo for weeks. Why was it failing now?

I checked the git diff. The billing fix was a one-line Python change. No RBAC touched. But the PreSync Job scans the entire rendered manifest tree, not just the diff. And kube-radar had a new rule I had merged the night before — version v0.3 — that now flagged pods/exec as critical, not just a warning.

The rule change was good. The ServiceAccount was overprivileged. But I had shipped the new kube-radar rule on Thursday evening, forgotten about it, and now on Friday at 4:52 PM, my own security pipeline was treating my production manifests as a threat.

ArgoCD showed SyncFailed.

The billing fix was not deployed.

4:55 PM: The Panic Override

I had two choices:

Option A: Fix the RBAC properly — create a restricted Role with only the necessary permissions, update the ServiceAccount binding, run the tests, commit, push, wait for the pipeline.

Estimated time: 20 minutes. It was 4:55 PM. The invoice batch job ran at 5:00 PM.

Option B: Bypass the security pipeline and force the sync.

ArgoCD lets you do this. You can click "Sync" with "Prune" and "Replace" checked. You can skip the PreSync hook. I am an admin. I have the power.

I hovered over the button.

And I realized: if I bypassed my own security gate the first time it inconvenienced me, the entire system was theater. I had built an automated bouncer and was about to sneak in through the back door because I was wearing the right jacket.

So I didn't.

5:03 PM: The Real Fix (And The Bug I Actually Shipped)

I spent 8 minutes — invoice job be damned — writing a proper Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: vehiclemetrics-billing
  namespace: vehiclemetrics-prod
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]

Removed pods/exec. Removed the ClusterRole binding. Applied the principle of least privilege.

Committed. Pushed. Pipeline passed. ArgoCD synced at 5:02 PM.

The invoice job? It ran at 5:03 PM. With the correct timezone fix. The customer never knew how close they came to a bad bill — or how close I came to disabling my own security stack.

But here's the part that haunts me:

The original pods/exec permission wasn't even needed. It was a copy-paste from a debugging session three weeks ago. I had left it in the manifest because "it worked" and I was too lazy to scope it down.

My security pipeline didn't create a problem. It revealed a problem I had been ignoring. The thing that blocked my salary was the thing that was already wrong.

The Aftermath: 48 Hours of Paranoia

I spent the weekend reviewing every manifest in the repo. Here's what I found:

Four critical issues. In my own repo. That I had written or approved.

Without the automated gate, they would have stayed there until someone exploited them — or until a compliance audit found them and I had to explain why a billing service could exec into pods.

What I Changed (The No-Escape-Hatch Problem)

The system was right to block me. But the system was also dangerous because it had no emergency override that didn't require me to become a liar.

Here's the architecture now:

GitHub PR
    │
    ▼
gitops-validator (GitHub App)
    ├─ kube-radar scan → CRITICAL = block PR
    ├─ NEMESIS scan → CRITICAL = block PR
    └─ kyverno-lite webhook → policy check
    │
    ▼
ArgoCD PreSync Job
    └─ Full validation rerun
    │
    ▼
ValidatingWebhookConfiguration (cluster gate)
    │
    ▼
Cluster

The fix: I added a @security-override label. If a PR is labeled with this, the pipeline still runs, still reports every violation, but emits a warning instead of a block. The label can only be applied by a GitHub Team called security-admins, which has exactly one member: me. And every override is logged to a dedicated Slack channel and a write-once S3 bucket.

I also changed failurePolicy: Fail to failurePolicy: Ignore on the admission webhook, with a twist: if the webhook is unreachable, ArgoCD flags the Application as Unknown and pauses automated syncs. Security is enforced when healthy. Availability is preserved when degraded.

webhooks:
  - name: rbac.gitops.lebeltagy.io
    # ...
    failurePolicy: Ignore  # Don't crash deploys if webhook is down

But the real fix wasn't technical. It was procedural:

I stopped treating "works" as the standard. "Least privilege" is the standard.

The Numbers

What I'd Do Differently

1. Don't ship new scanner rules on Thursday night

If you're changing what "critical" means, do it Monday morning when you have the week to deal with the blast radius. Not the day before you might need to deploy a hotfix.

2. Every guardrail needs a documented escape hatch

Not a secret backdoor. A visible, audited, tightly-scoped bypass. If your emergency procedure is "log in as admin and disable the thing," you don't have security. You have security theater with an intermission.

3. Scan the diff, not the world

The PreSync Job originally scanned the entire manifest tree. Now it scans only the Helm release diff: what changed, not what exists. Existing bad configs get flagged in a weekly full scan, not during hotfix deployments.

4. Your production manifests are dirtier than you think

I promise you. Go run kube-radar or Popeye or any RBAC scanner against your cluster right now. You'll find something embarrassing. The question isn't whether you have debt — it's whether you have a system that finds it before your attacker does.

Why I'm Keeping It

It's been three weeks since the Friday incident. The Dutch pilot expanded to a second customer. I haven't had a single 3 AM security scare. And when my latest contributor opened a PR with pods/exec in it, the pipeline blocked it before I even saw the notification.

I didn't have to be the bad cop. The code was.

Is automated GitOps security more work to maintain? Yes. I spend maybe an hour per week tuning rules and reviewing override logs.

Is it worth it? Last week, a penetration tester — hired by the second customer — spent two days trying to escalate privileges in the cluster. He found one over-scoped Role. It was in a staging namespace with no production data.

He wrote in his report: "The target environment exhibits unusually robust RBAC hygiene for an early-stage SaaS platform."

That sentence was worth every minute.

TL;DR — The "Don't Block Your Own Salary" Checklist

• [ ] Run an RBAC scanner against your cluster today (kube-radar, Popeye, or rbac-audit)
• [ ] Add a PreSync security gate to ArgoCD/Flux before your next deploy
• [ ] Ship new scanner rules on Monday, not Thursday
• [ ] Build an override mechanism that is audited, not secret
• [ ] Change failurePolicy: Fail to Ignore if you don't have 24/7 webhook SREs
• [ ] Scan the diff for deploy gates, scan the world for weekly audits
• [ ] Remember: the pipeline that blocks you is the pipeline that saves you

Want the admission webhook code + ArgoCD PreSync manifests? Drop a comment — I'll open-source the gitops-validator repo if there's interest.

Ever been locked out of your own system by your own automation? Tell me your war story below. We can start a support group.

Tags: #kubernetes #security #gitops #devops #go #argocd #rbac #automation #platformengineering