Note: Names, identifiers, infrastructure paths, dates, and selected implementation details have been generalized or rounded to protect confidential information. The architecture, failure modes, engineering decisions, and outcomes are real.
Table of Contents
- The Cleanup Nobody Wanted to Own
- Why Deleting an Account Was Not a Delete
- The Plan: Start Small, Learn, Then Scale
- The First Warning: A Database Under Pressure
- The Hidden Side Effects
- The Silent Telemetry Failure
- The Account That Would Not Finish
- The Independent Review That Changed the Design
- How AI-Native Operations Changed the Execution Model
- The Outcome
- What I Would Build Differently
The Cleanup Nobody Wanted to Own
The request sounded simple:
Retire a large population of inactive tenants that had accumulated over several years.
They included abandoned trials, expired subscriptions, and dormant free accounts. They consumed storage, increased backup volume, complicated retention compliance, and left millions of related records distributed across the platform.
The approved scope covered more than 120,000 tenants and approximately 1.4 million user records.
I understood the operational goal and the risks, but I was not the author of the legacy Rails application or an expert in every callback hidden inside its account-deletion path. I also could not pause normal production activity while studying the codebase for months.
What I did have was:
- Platform and infrastructure fundamentals
- Access to subject-matter experts for review and approvals
- An AI operations environment connected to source control, observability, cloud infrastructure, ticketing, and automation systems
That combination changed what was possible.
Why Deleting an Account Was Not a Delete
At first glance, account retirement looked like a database operation. In reality, one model-level delete initiated a distributed workflow:
Retire tenant
├── Remove user-dependent records
├── Disable application provisioning
├── Run model destruction callbacks
├── Generate audit events
├── Publish asynchronous cleanup messages
├── Notify dependent services
├── Remove remaining orphan records
└── Record progress and verification evidence
A path that is acceptable for one interactive deletion can become dangerous when repeated continuously at scale.
The important engineering question was not:
How quickly can I delete rows?
It was:
Which synchronous and asynchronous side effects will amplify when this operation runs thousands of times?
The Plan: Start Small, Learn, Then Scale
I divided the approved work into risk-based phases rather than attempting one enormous execution.
- Pilot phase — a few hundred low-risk accounts
- Expanded trial phase — a few thousand accounts
- Bulk low-risk phase — tens of thousands of dormant trial accounts
- Free-account phase — another large population with additional validation
Higher-risk terminated and paid-account populations remained outside the approved scope pending further business review.
The execution tooling included:
- CSV-driven account selection
- Dry-run mode enabled by default
- Preflight checks
- Bulk cleanup of high-volume dependent rows
- Per-account checkpoints
- An append-only operational audit trail
- A circuit breaker for consecutive failures
- Progress metrics and events
- A resumable batch runner
- Independent post-run database verification
The pilot completed cleanly. That gave me confidence to proceed.
Then the second large execution exposed what the pilot could not.
The First Warning: A Database Under Pressure
During an early production batch, an internal database alert fired. CPU on the events datastore had climbed to a critical level and remained elevated.
There was no customer-facing outage, but the signal was serious enough to stop and investigate.
The database team identified a family of expensive range-scan queries. I compared the account identifiers in those queries with the active cleanup batch.
Every sampled identifier matched.
The deletion process was not directly querying the events database, so the immediate question was:
What did the account-deletion path trigger indirectly?
That question took me into the legacy callback chain.
The Hidden Side Effects
I found two separate paths affecting the events datastore.
Path 1: Audit-event generation
Deleting an account cascaded into users, roles, directories, and other objects. Multiple callbacks generated audit records during that cascade.
One deletion might produce a manageable amount of work. Thousands of continuous deletions created a sustained write workload and fed additional downstream processing.
Path 2: Asynchronous historical-event cleanup
A model callback also published an asynchronous message requesting historical-event deletion for each retired account. A downstream worker consumed those messages and performed large range scans.
The main cleanup process appeared healthy while the message queue continuously generated expensive work elsewhere.
That distinction mattered:
Visible process: account retirement is progressing normally
Hidden process: asynchronous workers are saturating another database
The controlled mitigation
For the approved low-risk phases, I suppressed these two side effects inside the one-shot cleanup process only, after review with the relevant stakeholders.
Conceptually:
# Simplified example — not production source
class AuditWriter
def self.record(*args)
# Suppressed only inside this one-shot maintenance process
end
end
class Tenant
def enqueue_historical_event_cleanup
# Deferred to a separately controlled database cleanup
end
end
The patches did not modify the deployed application, its containers, or other workers. They existed only inside the Ruby process executing the maintenance task and disappeared when that process exited.
I preserved a separate operational audit trail and created a tracked follow-up for controlled removal of orphaned historical events during maintenance windows.
The result: events-database utilization returned to a stable baseline while the tenant cleanup continued.
The Silent Telemetry Failure
The next failure was quieter.
The cleanup logs showed accounts being retired successfully, but the progress dashboard showed no data. The task also claimed that metric publishing had completed.
The problem was an API-version mismatch. The code sent a payload shaped for one metrics API to another version. The monitoring service returned an HTTP client error, but the implementation only treated exceptions as failures. Since the HTTP library returned a response object instead of raising, the task reported success.
Work completed: yes
Metric request transmitted: yes
Metric accepted: no
Exception raised: no
Agent narrative: progress appears normal
Independent dashboard: no data
The dashboard's disagreement with the task logs exposed the issue.
I corrected the payload and, more importantly, changed the design rule:
A successful network call is not a successful outcome. Parse the response, validate the result, and read back the effect from an independent system.
This became one of the most important lessons from the entire operation.
The Account That Would Not Finish
A later batch encountered an extreme outlier.
The planning spreadsheet showed only one enabled user, but the live database contained an unexpectedly large population of historical user records plus a large association graph.
The high-volume user rows were removed, but the remaining model cascade did not finish. The process continued consuming CPU without crashing, exiting, or producing new logs.
From the outside, the runner looked dead.
Someone started a replacement runner. The original process was still alive.
For an extended period, one runner was stuck while another continued healthy work.
The miss was surprisingly mundane: the process search looked for the rake-task name, while the operating system displayed the Ruby executable. The stopped log was interpreted as evidence that the process had died.
The prevention controls were straightforward:
# Simplified guardrail
Timeout.timeout(max_account_seconds) do
retire_account(account)
end
# Simplified single-instance guard
if another_runner_exists; then
echo "Another runner is active; refusing to start."
exit 1
fi
I also added:
- A PID lockfile
- Process-tree checks
- Live database row-count validation for outliers
- A hard threshold requiring separate review for exceptionally large accounts
- A heartbeat/progress-age monitor so silence becomes an alert
The Independent Review That Changed the Design
I shared the implementation with a senior architecture reviewer, who used a separate AI-assisted review process to challenge the design.
The review identified issues that had been easy to miss under operational pressure:
- A credential supplied in an unsafe way
- A database timeout setting with a broader scope than intended
- Missing transactional boundaries around per-account work
- Compliance implications of suppressing normal audit events
- The need to assign ownership and deadlines to deferred cleanup
- Risks in a partial method stub
- The absence of proof that process-local patches had actually loaded
- Poor visibility of behavior-changing patches in startup logs
I addressed the findings by:
- Rotating and relocating credentials to managed configuration
- Limiting database settings to explicit transactions
- Adding per-account transactional protection where safe
- Documenting the approved alternate audit trail
- Creating a separately owned database cleanup task
- Making the stub defensive against interface changes
- Verifying patched method source locations at startup
- Printing a prominent banner showing every process-local override
The value of the second AI review was not that it was automatically correct. It generated a rigorous second set of questions for me and the relevant specialists to evaluate.
How AI-Native Operations Changed the Execution Model
I understood infrastructure, operational risk, and what a safe outcome needed to look like. I was not a specialist in the legacy application's Ruby internals or every API involved in the workflow.
AI bridged that specific knowledge gap.
Building the first version
I described the operational contract:
Read account identifiers from a controlled input, default to dry run, checkpoint after each item, stop after repeated failures, emit progress, and preserve an audit trail.
The agent produced a first implementation. I reviewed the workflow and validated it with application owners before execution.
Investigating the database signal
I asked the agent to compare database evidence with the active cleanup population, trace the relevant callback paths, and explain how a foreground delete could trigger work in another datastore.
It helped connect application code, queue behavior, and database telemetry into one causal chain.
Debugging missing telemetry
I described the disagreement between successful cleanup logs and an empty dashboard. The agent inspected the metrics-publishing code, compared it with the API contract, and identified the payload mismatch.
Diagnosing the hung execution
I asked it to inspect process state rather than rely on the stopped log. It helped identify the still-running child process and generated the restart and single-instance protections.
Operating the long-running job
Throughout the execution, the same session could answer:
- How many accounts are complete?
- Has progress stalled?
- Is database utilization stable?
- Did a batch produce errors?
- What is the estimated completion window?
- Which follow-up tasks require another team?
It queried the automation system, observability platform, ticketing system, and cloud environment without requiring me to switch among multiple consoles.
This is the broader AI-native operations pattern:
- A human defines the goal, constraints, and acceptable risk.
- The agent gathers live evidence and proposes implementation steps.
- Deterministic controls constrain execution.
- Independent systems verify the result.
- Humans review exceptions and authorize higher-risk phases.
That pattern is transferable to migrations, cost optimization, compliance audits, incident response, and many other operational workflows.
The Outcome
Across the four approved phases:
| Outcome | Result |
|---|---|
| Dormant tenants retired | More than 120,000 |
| Associated user records removed | Approximately 1.4 million |
| Execution duration | Under two weeks |
| Customer impact | None observed |
| Customer-facing incidents | None |
| Independent verification | Completed against the production database |
| Higher-risk phases | Held for additional business approval |
The project also left behind reusable operational capabilities:
- A resumable maintenance-runner pattern
- Independent verification practices
- Process-local safety checks
- Better telemetry validation
- A clear owner and process for deferred historical-data cleanup
- A stronger review model combining AI analysis with human architecture oversight
What I Would Build Differently
1. Make independent verification part of version one
The layer doing the work should not be the only layer declaring success. Every important action needs a deterministic read-back from the system it changed.
2. Treat callbacks as distributed architecture
A model callback can publish messages, call external services, and create work in another database. At scale, callback analysis is capacity planning.
3. Validate live cardinality, not spreadsheet summaries
An account with one enabled user may still contain tens of thousands of historical rows. Selection data is a planning snapshot; the production database is ground truth.
4. Detect absence of progress
Errors are not the only failure signal. A process can remain alive while accomplishing nothing. Track heartbeat age, last completed item, and work rate.
5. Keep high-risk operations separable
Retiring tenants and purging years of historical events did not need to happen in the same transaction or maintenance window. Separation made both safer.
6. Use AI to extend expertise, not bypass it
AI made it possible to move quickly through unfamiliar application internals. The project remained safe because experienced engineers defined the constraints, challenged outputs, and verified results independently.
The surprising part was not that AI could generate Ruby or shell commands.
The surprising part was that one AI-integrated operations session could maintain context across application code, database telemetry, automation logs, cloud infrastructure, and project tracking for the duration of a complex production program.
That cross-signal continuity was the real force multiplier.
I'm Vinothsingh Elumalai, a Platform Engineering leader building AI-native operations at enterprise scale. I write about using AI to extend engineering expertise across unfamiliar systems while keeping deterministic controls and humans in the loop.
This article is part of my AI-Native SRE series.
