The alert was one line:
checkout-service: requests failing shortly after a release
I ran it through the same LLM twice.
Without memory, I got six plausible causes: code regression, config drift, dependency changes, schema mismatch, resource exhaustion, and traffic routing.
With memory, I got this:
Likely root cause: a missing or misnamed environment variable.
Historical context: the same pattern appeared in a previous incident.
Warning: do not scale replicas because that mitigation failed before.
Recommended fix: roll back and correct the Helm value.
Previous resolver: Priya Nair.
The model didn't get smarter.
It got experience.
What I built
OnCall Memory is an incident triage agent for software engineering teams.
The history belongs to a fictional payments company, Northwind Payments, with eight services across Java, Go, Python, and Node, backed by Postgres, Redis, Kafka, and Elasticsearch.
The data is synthetic: 40 past incidents drawn from six failure families:
- exhausted database pools
- Redis eviction stampedes
- bad configuration after a release
- expired certificates
- Kafka poison messages
- full disks
The important design choice is that the same underlying failure can appear with different symptoms on different services.
The team's experience matters more than matching exact words.
Why persistent memory matters
An incident is more than a document.
I wanted the agent to remember:
- which fix worked
- which fix failed
- who solved the incident
- what happened in earlier incidents
- what should be avoided next time
That is why the project uses Hindsight as the persistent memory layer.
The architecture is simple:
text
Current Alert
↓
Hindsight Recall
↓
Historical Experience
↓
LLM Reasoning
↓
Operational Recommendation
↓
Resolved Incident
↓
Post-mortem retained as new memory
