The most common question a litigator asks before walking into court is five words long: "What happened last time?"
It sounds like the easiest question an agent with memory could get. It turned out to be the one that forced me to think hardest about time. When I first wired up the memory layer, "last time" quietly meant "the file you uploaded most recently." Those are different things, and in a law practice the gap between them can be eighteen months.
The setup
I built Tareekh, a memory for a litigator's practice. The lawyer photographs pocket-diary pages, uploads certified order sheets, drops in deeds and notices, or types a quick note. Tareekh reads each upload with Gemini on Vertex AI, splits it into one entry per hearing, files each entry under the right case, and retains it in Hindsight, an open-source memory engine for agents. An agent answers questions from that memory and cites its sources.

Figure: Tareekh's architecture. Each hearing becomes one Hindsight memory in a single bank per lawyer.
The practice I test with belongs to Adv. Aditya Varma and his junior Divya. There are five cases and 79 uploads covering April 2025 to September 2026.
How lawyers actually enter data
Nobody uploads a hearing on the day it happens. The real pattern is a backlog. A lawyer who starts using the tool has a year of diary pages in a drawer, so they photograph forty of them one evening. After that, uploads come in bursts: a week of notes on a Sunday, an order sheet when the certified copy finally arrives three weeks after the hearing, a scanned deed from 2011 when a dispute about it comes up.
My backlog loader makes this worse, and makes the problem impossible to miss. It sends the uploads oldest first, but it runs five workers in parallel because each upload spends most of its time waiting on OCR:

scripts/load_backlog.py

several at once: each upload waits mostly on the OCR model, not on this machine

with ThreadPoolExecutor(a.workers) as pool:
for i, (up, m, r) in enumerate(pool.map(one, manifest), 1):
...
So the order in which entries reach Hindsight is roughly chronological, with the details decided by whichever OCR call finishes first. If a memory's time is "when it was stored," the timeline is noise.

Figure: Stamped at upload time, 79 uploads land on one evening and "last time" means whichever file finished last. Stamped with the hearing date, they spread across eighteen months and "last time" means the actual last hearing.
What Hindsight does with time
Hindsight treats time as a first-class part of a memory. Every item you retain can carry a timestamp, and recall results come back with occurred_start, the time the fact is about. Recall runs a temporal path alongside semantic and keyword search, so a query that implies a time ("last hearing", "in June") can lean on it. That's exactly what you want, as long as the timestamp means what you think it means.
If you don't pass one, the reasonable default is "now." For a chat assistant that's correct. For a legal record it's wrong in a subtle way: nothing errors, the answers look plausible, and they're about the wrong hearing.
The fix: the hearing date is the timestamp
Every Tareekh memory is stamped with the date of the hearing it describes. The rule is short enough to live in a comment:

memory.py, build_item() (trimmed)

cid, date = entry["case_id"], entry["hearing_date"]
return {
"content": header + "\n" + entry["text"],
"timestamp": f"{date}T10:30:00+05:30", # the HEARING date, never the upload time
"metadata": {"source_file": source_file, "upload_id": upload_id,
"case_id": cid, "hearing_date": date, "author": author},
"document_id": f"{upload_id}:{cid}:{date}",
}
10:30 IST is roughly when district courts start sitting, so every hearing gets the same plausible time of day. The date also goes into metadata, because metadata comes back with every recalled fact and that's what the UI shows on a citation chip. And it's part of document_id, so uploading the same diary photo twice replaces the memory rather than duplicating it.
The hard part is getting the date right, not storing it.
Where the hearing date comes from
A diary page prints the date at the top. An order sheet has rows like 22.06.2026: Counsel for defendant again sought time.... A typed note might say "12/9 Gorle partition" and nothing else. And almost every note contains a second date that is not the hearing date: the next date.
The segmentation prompt is explicit about that trap:

ingest/segment.py (excerpt from the system prompt)

• hearing_date: the date the hearing happened (YYYY-MM-DD). Diary pages print it at the top; order sheets use dd.mm.yyyy. Dates like "next date 5 Oct" are NOT the hearing date. If unknown, use null. When the model returns null, or something that doesn't parse, a plain-code repair step falls back to the filename. Phone cameras name files IMG_20250618_114747.jpg, so the photo's own name usually knows the day it was taken. That's usually the hearing date, or the evening after it: DATE_IN_NAME = [ (re.compile(r"(20\d{2})(\d{2})(\d{2})"), lambda m: f"{m[1]}-{m[2]}-{m[3]}"), # IMG_20260812... (re.compile(r"(20\d{2})-(\d{2})-(\d{2})"), lambda m: f"{m[1]}-{m[2]}-{m[3]}"), # court_notes_2026-08-12 ] ... date = _valid_date(e.get("hearing_date")) if not date and fallback_date: date = fallback_date conf = min(conf, 0.75) reasons.append("date from filename/hint") The confidence cap matters. Auto-confirm needs 0.8, so a date guessed from a filename always goes to a human for a quick look before it becomes a memory. A wrong timestamp is the kind of error nobody notices until it matters.

Figure: A diary page from the test practice. The date is printed at the top, and the file is named IMG_20250618_114747.jpg, which agrees with it.
One exception: things said in chat
Tareekh also remembers durable things the lawyer says in chat: "we won't settle for Srinivas's half only," "insist on 18% interest from June 2023." Those are stamped with the moment they were said, because that's when the decision was made. They're labelled CHAT MEMORY (... not a court record) in their content and tagged type:chat_memory, and the agent ranks them below records and notes. Two kinds of time, stored deliberately, both in the same bank.
Before and after
Take the Seabreeze specific-performance suit. Its most recent hearing is 24 September 2026, recorded in Divya's typed note ("objections to IA 1187/2026 (amendment): draft 60% done"). But the most recently uploaded file touching that case could easily be an older order sheet whose certified copy took weeks to arrive, or the 2024 agreement of sale that someone scanned last.
With upload-time stamps, "what happened last time in Seabreeze SP suit?" leans toward whatever file finished processing last. The answer is fluent, it's cited, and it describes the wrong day. That combination is the worst possible failure for a tool a lawyer reads while standing in court.
With hearing-date stamps, the Today page and the agent agree. The cause-list card shows "Previous hearing, 24 Sept · Divya's note," and the answer cites that note. The case_timeline tool sorts by the same hearing_date in metadata, so asking for the history of a case gives you the hearings in the order they happened, whatever order you uploaded them in.

Figure: The Today page. Each listed matter shows its previous hearing by hearing date, not by upload date.
What I'd pass on
Decide what time a memory is about before you store the first one. "When it happened" and "when I learned it" are both valid, but you have to pick on purpose. Hindsight gives you the field; it can't guess which one you meant.
Put the date in three places. In the timestamp so retrieval understands time. In metadata so every citation can show it. In the document_id so re-uploads are idempotent per hearing. Batches get re-run, and without that you pile up duplicates.
Teach the extractor which dates to ignore. Next dates, filing dates, dates in the body of a deed: a legal note is full of dates, and exactly one of them is the hearing. A single sentence in the prompt did more than any post-processing.
Cap confidence on guessed dates. A fallback is fine. A fallback that skips human review is how a wrong date becomes a confidently wrong memory.
It's not perfect. A photo taken the morning after a hearing gets the next day's date from its filename, and a lawyer who photographs a week of pages on Sunday gets seven entries dated Sunday unless the page header is readable. The review screen catches most of this. It won't catch all of it.
If you're adding memory to an agent that works over records with their own dates (tickets, medical notes, incidents, transactions), this is probably the first design decision to get right. The Hindsight docs cover retain timestamps and how recall uses them, and Vectorize's overview of agent memory is a good read on why "when" is part of what a memory is, not an afterthought.