85 containers. 24 PostgreSQL databases. 67 domains. 232 cron jobs. One developer. 120 EUR/month in Hetzner bills.
This is not a startup fantasy pitch. This is my production infrastructure for a SaaS ecosystem serving German golf clubs, a golf school management platform, a community platform, a CRM, and an auth service. Every customer gets their own database. Physical tenant isolation, not software filters.
People tell me this cannot work. The containers disagree.
The Stack
Next.js for all frontends. Single-tenant PostgreSQL per customer (Supabase stacks). Docker on bare metal. Coolify for deployment orchestration. Traefik as the reverse proxy handling 67 domains. Two Hetzner servers in Germany. Total infrastructure cost: 120 EUR/month.
The single-tenant architecture is a deliberate trade-off. Multi-tenant saves infrastructure cost, but one RLS bug exposes every customer's data. One compromised tenant enables lateral movement to all others. GDPR Article 17 deletion in multi-tenant requires complex cross-tenant queries. In single-tenant, deletion is DROP DATABASE. No residual risk.
The cost is more operational complexity. Which is exactly why automation is not optional.
176 Guard Rules: The Immune System
My AI agents (Claude Code with custom hooks) execute roughly 80% of daily development and operations work. That is dangerous without constraints. So I built a guard system: 176 shell scripts that fire on every command, every file edit, every session end.
The architecture is simple. Four dispatchers route to context-specific guards:
#!/bin/bash
# Pre-Bash-Dispatcher: Loads guards based on command profile.
# Not all 176 guards fire on every command. Profiling classifies
# each command (git, docker, npm, database, deploy, comms) and
# loads only relevant guards.
set -uo pipefail
GUARDS_DIR="$(dirname "$0")/guards"
INPUT=$(cat)
CMD=$(echo "$INPUT" | jq -r '.tool_input.command // ""')
# 8 security gates fire ALWAYS, non-negotiable:
# tabu-gate, pii-gate, api-key-guard, secret-output-guard,
# pre-exec-file-scanner, gate-file-guard (guards protect themselves),
# agent-control-policy, main-push-guard
PROFILE=$(classify_command "$CMD") # git|docker|npm|database|deploy|...
for guard in "$GUARDS_DIR/$PROFILE"/*.sh; do
result=$("$guard" "$CMD" "$SESSION_ID")
if echo "$result" | jq -e '.permissionDecision == "deny"' > /dev/null 2>&1; then
echo "$result"
exit 0
fi
done
The guards protect against real problems I have encountered: agents pushing directly to main, leaking PII into logs, deleting production containers, skipping pre-mortem checks before destructive operations, or committing API keys.
96% of all rules (83 out of 86) are enforced automatically. The remaining 3 require human judgment. No governance document that nobody reads. Executable rules that block before damage happens.
The Self-Healing Watchdog
Containers disappear. Coolify deployments fail silently. Traefik loses backend connections. At 85 containers, something breaks every week.
The watchdog runs every 5 minutes via cron and restores service from the last known good state:
#!/bin/bash
# live-app-watchdog.sh
# Detects missing Coolify containers, restores from last local image.
# No secrets written to logs.
set -uo pipefail
LOG="/var/log/live-app-watchdog.log"
STATE_DIR="/var/run/live-app-watchdog"
APPS=(
"golf-club-community|golfclub-app.de"
"golfschul-app|golfschul-app.de"
"golf-auth-provider|auth.golfclub-app.de"
# ... 12 more entries
)
for entry in "${APPS[@]}"; do
IFS='|' read -r name domain <<< "$entry"
container=$(docker ps -q --filter "name=$name" 2>/dev/null)
if [ -z "$container" ]; then
log "MISSING: $name ($domain)"
http_code=$(curl -sS -o /dev/null -w "%{http_code}" \
"https://$domain/api/health" --max-time 5 2>/dev/null)
if [ "$http_code" != "200" ]; then
last_image=$(docker images --format '{{.Repository}}:{{.Tag}}' \
| grep "$name" | head -1)
if [ -n "$last_image" ]; then
docker run -d --name "${name}-emergency" \
--network coolify "$last_image"
notify_once "critical" "Emergency container started" \
"$name restored from $last_image" "$name-restore"
fi
fi
fi
done
Emergency containers are temporary. The watchdog notifies me via ntfy.sh push notification, and the next Coolify deployment replaces the emergency container with a proper one. The point is: the customer never notices.
The Crystallization Loop: Mistakes Become Permanent Rules
This is the mechanism that makes the system improve without me writing new rules. When an AI agent makes a mistake, the learning gets captured. When that learning proves useful across 3+ sessions with a quality score of 4 or higher, it crystallizes into a permanent rule.
#!/bin/bash
# auto-skill-crystallizer.sh (runs at session end)
# Sessions with >10 tool calls get analyzed for patterns.
# Writes proposals, never creates rules autonomously.
INPUT=$(cat 2>/dev/null)
SESSION_ID=$(echo "$INPUT" | jq -r '.session_id // "default"')
TOOL_CNT=$(cat "/tmp/claude-toolcount/$SESSION_ID" 2>/dev/null || echo 0)
# Only analyze substantive sessions
[ "${TOOL_CNT:-0}" -lt 10 ] && exit 0
# Max 1 proposal per day to prevent spam
VORSCHLAG_FILE="$VORSCHLAG_DIR/$(date +%Y-%m-%d)-crystallizer.md"
[ -f "$VORSCHLAG_FILE" ] && exit 0
# The crystallization criteria:
# Score >= 4 AND Runs >= 3 -> learning becomes permanent rule
# Score 5 + confirmed by feedback -> immediate crystallization
MSG="Crystallizer: Session had ${TOOL_CNT} tool calls. "
MSG+="Check: (1) Recurring pattern no skill covers yet? "
MSG+="(2) New solution that would help future sessions? "
MSG+="If yes, write proposal. If no, ignore."
The numbers after 18 months: 211 rules crystallized from agent experience. Not written by a human. Distilled from 1,448 autonomous tasks. 1,087 completed successfully. 88% success rate.
The crystallization loop is the core of what I call the GRIP framework (Guards, Resilient, Isolated, Public). Guards prevent mistakes. When mistakes happen anyway, the resilience loop turns them into new guards. Isolation limits the blast radius. Public transparency makes everything auditable.
What 1,087 Autonomous Tasks Actually Means
My agents handle deployments, database migrations, backup verification, security patching, content generation, monitoring, and customer support triage. The 232 cron jobs include 6-hourly Supabase permission heals, daily graph rebuilds, backup rotation, certificate renewals, health checks, and content pipeline automation.
The 88% success rate means 12% of tasks need human intervention. That is honest. Agents break things. The guard system catches most of it before production impact. The crystallization loop ensures the same failure mode rarely happens twice.
The stop dispatcher at session end blocks the agent from exiting if it has uncommitted code, unverified deployments, or unwritten documentation. The agent cannot just walk away from unfinished work.
The Honest Limitations
Single-tenant architecture means I provision infrastructure per customer. At 24 databases, this is manageable. At 240, I will need automation I have not built yet.
The guard system adds latency. Every bash command passes through the dispatcher before execution. On a hot day with a large command, that is 200ms of overhead.
Some crystallized rules conflict with each other. A rule saying "always run tests before deploy" conflicts with "emergency containers must be started within 60 seconds." Conflict resolution is still manual.
And 120 EUR/month only works because I am the only developer. The moment I need to onboard someone, the operational complexity becomes a liability, not an advantage.
Why I Wrote This Down
I spent 18 months building this system through daily practice, not through planning. The guard system did not start with 176 rules. It started with 3, after an agent pushed directly to main at 2 AM.
I documented the entire approach in a book because the principles (executable governance, learning from agent mistakes, deliberate isolation) apply far beyond my specific stack. If you are running AI agents in production, you need something like this. Not necessarily my implementation. But the pattern.
Get the book: Paperback ($24.99) https://amazon.com/dp/B0HDMVKRMG | E-Book ($9.99) https://amazon.com/dp/B0HDMK7QJ1
