When a build breaks, the bug fixes itself

When a build breaks, the bug fixes itself

We stopped babysitting CI failures. Now a red build files its own bug — and an AI agent picks it up and ships the fix.

PROBLEM — A failed build told no one

Our CI would fail, and then… nothing would happen. The failure sat quietly in a build console that nobody keeps open. Eventually someone would notice a change hadn't gone out, go digging, and realize the build had been red for hours.

And noticing was the easy part. Actually resolving it meant a whole code session: pull up the logs, find the failing step, reproduce it, and have an engineer sit down and personally shepherd the fix from broken to green. Every red build cost real human hours — plus the invisible tax of the delay before anyone even knew there was a problem.

The true cost of a broken build was never the build. It was a person having to find it, understand it, and hand-fix it.

SOLUTION — The failure files its own ticket — and an agent takes it from there

Now nobody watches a console and nobody triages. The moment a build fails, it automatically files a bug in Shipeasy — our ops platform — as a real, prioritized ticket with the failing step, the branch, and a link to the logs already attached.

From there it leaves human hands entirely. Shipeasy hands the bug to an AI agent, which investigates the failure, writes the patch, and opens a pull request against it. The loop that used to be "human notices → human reads logs → human fixes" is now "build fails → bug appears → agent fixes." The engineer's job shrank to reviewing a PR that already exists.

DESIGN — How the whole thing hangs together

The pipeline is deliberately boring — every hop is either something the cloud already does for free, or a service we already run:

• Cloud Build — build fails: a red deploy on main publishes automatically
• Pub/Sub topic → push subscription: filters to FAILURE · TIMEOUT · INTERNAL_ERROR
• HTTPS POST /webhooks/cloud_build
• Webhooks::CloudBuildController: verify token · decode · dedupe by build id
• ShipeasyOps::Client#file_bug: POST /api/admin/ops (type: "bug")
• Shipeasy ops queue — bug filed: fans out to GitHub issue + Slack
• ✦ AI agent investigates → opens a PR

What makes this cheap is the shape of it: we added no new infrastructure. The event was already on a bus (Pub/Sub). We already ran a service that could receive it. All we wrote was the glue in the middle.

IMPLEMENTATION — How it worked out in Rails

The build side needed no changes at all — Cloud Build publishes to the cloud-builds topic on its own. So the work was three small pieces: one gcloud command, one route, and one controller.

1. Point a filtered push subscription at the app. Pub/Sub does the delivery; the filter means the endpoint only ever wakes for a real failure.

gcloud pubsub subscriptions create cloud-build-failures \
  --topic=cloud-builds \
  --push-endpoint="https://our-app/webhooks/cloud_build?token=$SECRET" \
  --message-filter='attributes.status = "FAILURE"
    OR attributes.status = "INTERNAL_ERROR"
    OR attributes.status = "TIMEOUT"'

1. Add the route — it slots into the same webhook surface as our other providers.

scope "/webhooks", module: :webhooks do
  post "cloud_build", to: "cloud_build#create"
end

1. The controller. It authenticates the push, decodes the build payload (it arrives base64-encoded in message.data), throws away anything that isn't a real failure, dedupes — Pub/Sub delivers at-least-once, so a redelivery must not file a second bug — and files the ticket.

class Webhooks::CloudBuildController < Webhooks::ApplicationController
  before_action :verify_token

  FAILURE_STATUSES = %w[FAILURE INTERNAL_ERROR TIMEOUT]

  # POST /webhooks/cloud_build
  def create
    message = params[:message]
    return head(:bad_request) if message.blank?

    build  = decode_build_json(message[:data])   # base64 JSON → Hash
    status = (message.dig(:attributes, :status) || build["status"]).to_s
    return head(:ok) unless FAILURE_STATUSES.include?(status)

    return head(:ok) unless first_delivery?(build["id"])   # dedupe

    file_bug_for(build, status)
    head :ok
  end

  private

  # Compare-and-set on the build id: the first delivery wins, redeliveries no-op.
  def first_delivery?(build_id)
    Rails.cache.write("cloud_build:filed:#{build_id}", true,
                      unless_exist: true, expires_in: 7.days)
  end

  def verify_token
    expected = App::Secrets.cloud_build_webhook_secret
    provided = params[:token] || request.headers["X-CloudBuild-Token"]
    head :unauthorized unless
      ActiveSupport::SecurityUtils.secure_compare(provided.to_s, expected.to_s)
  end
end

The one line that files the ticket. The controller hands the build context to a thin Shipeasy client. This is the exact call — it turns a red build into a first-class bug that opens a GitHub issue, pings Slack, and becomes eligible for the auto-fix agent:

def file_bug_for(build, status)
    subs = build["substitutions"] || {}

    ShipeasyOps::Client.new.file_bug(
      title:              "Cloud Build #{status.downcase} on #{subs["BRANCH_NAME"]} — #{subs["SHORT_SHA"]}",
      steps_to_reproduce: "Cloud Build trigger \"#{subs["TRIGGER_NAME"]}\" reported #{status}.",
      actual_result:      "Logs: #{build["logUrl"]}\n\n#{build.dig("failureInfo", "detail")}",
      expected_result:    "The build completes and deploys.",
      priority:           "high",
      tags:               %w[cloud-build ci],
    )
  end

And the client itself is just a typed wrapper over one HTTP call — no new gems, no framework. This is all it takes to create a bug on the platform:

def file_bug(title:, steps_to_reproduce:, actual_result:, expected_result:, priority:, tags:)
    post("/api/admin/ops", {
      type:             "bug",
      title:            title,
      stepsToReproduce: steps_to_reproduce,
      actualResult:     actual_result,
      expectedResult:   expected_result,
      priority:         priority,
      tags:             tags,
    })
  end

  def post(path, body)
    req = Net::HTTP::Post.new(URI("#{BASE_URL}#{path}"))
    req["Authorization"] = "Bearer #{@admin_key}"   # sdk_admin_… key
    req["X-Project-Id"]  = @project_id
    req["Content-Type"]  = "application/json"
    req.body = body.compact.to_json

    res = Net::HTTP.start(req.uri.host, req.uri.port, use_ssl: true) { |h| h.request(req) }
    JSON.parse(res.body)
  end

That's the whole integration. The failure travels from Cloud Build to a filed, prioritized bug through two managed hops and about forty lines of Ruby — and the moment it lands, it's the platform's problem, not a person's.

SHIPEASY — One problem, several ways to solve it

• Report it as an error via see(): Dead simple and already everywhere in our code — but it produces an auto-filed error, not a first-class bug with repro steps and priority. Great for exceptions; not quite the tracked work item we wanted.
• Public feedback ticket: The zero-auth path meant for in-app "report a problem" widgets. Perfect for user feedback, heavier than we needed for an internal signal.
• File a real bug via the admin API (chosen): A proper bug — title, repro, priority, tags — that immediately opens a GitHub issue, pings Slack, and is eligible for the auto-fix agent. Exactly the lifecycle a broken build deserves.

THE PAYOFF — The last red build nobody had to fix

A build died with FATAL ERROR: Reached heap limit — JavaScript heap out of memory. The bundle outgrew its memory ceiling. The agent recognized the OOM pattern and opened a PR with the entire fix:

- NODE_OPTIONS="--max-old-space-size=4096"
+ NODE_OPTIONS="--max-old-space-size=8192"

A teammate who doesn't write backend merged it. The person who "fixed" it never had to know what a heap limit is.

The failure notices itself, files itself, fixes itself, and asks a human for nothing more than a nod.

Before standing up new infrastructure to react to an event, check whether the event is already on a bus you can subscribe to — and whether something you already run can catch it.

The ops side of this runs on Shipeasy — the ticket, the agent, the rollout. Core product is the ops queue that delegates to agents. Free tier, no card required; Team at $49/seat/mo removes limits.

→ shipeasy.ai · → docs.shipeasy.ai/sdks/ruby · → github.com/shipeasy-ai/shipeasy