The Database I Deleted: Migrating an Agentic RAG App to AWS Serverless #2

Part 2 — the five decisions that fought back

Part 1 covered why an agentic RAG assistant — built 284 LLM Zoomcamp capstone submissions — moved off a single EC2 box onto AWS serverless, and the resulting architecture: CloudFront and S3 for a static SPA, one Lambda behind a shared-secret header, Bedrock for inference/embeddings/reranking, S3 Vectors and a SQLite artifact for retrieval, DynamoDB for state, Step Functions for ingestion. This part is the five places where the obvious design turned out to be wrong.

Quick recap of the shape: before, everything lived on one t4g.small — Postgres, a Streamlit process holding two torch models in RAM, minsearch indexes, Grafana — idling at ~$15/month. After, nothing idles: a Lambda behind CloudFront calls Bedrock for inference and DynamoDB/S3/S3-Vectors for state, and an offline Step Functions pipeline does the ingestion. Below is why each piece of the "after" ended up where it did.

Decision 1: I deleted the database

The obvious port of Postgres is Aurora Serverless v2 with the RDS Data API — HTTPS-based, so the Lambdas stay outside the VPC and you avoid a NAT gateway. I planned exactly that.

Then I priced the floor. Aurora Serverless v2 at a 0.5-ACU minimum is ~$43/month to sit there doing nothing — three times what the whole EC2 box cost. Dropping to a 0-ACU floor helps, but the auto-pause interval is 300 seconds minimum and the resume takes ~15 seconds, or 30+ after a day idle. For a portfolio demo visited twice a week, "30 seconds before the first byte" is the user experience.

So I looked at what the database actually held, and the answer was embarrassing:

278 projects and a few thousand library rows. 88 KB. Rebuilt from scratch by the ingestion pipeline, and read-only at query time.

That is not a database workload. That is a file.

The read path became a projects.sqlite artifact — the projects and libraries tables, the corpus, and an FTS5 index over it — built by the pipeline, published to S3, and pulled into the Lambda's /tmp on cold start. The write path (conversations, feedback, the spend ledger) went to DynamoDB. ~$43/month → ~$1/month.

The part I didn't expect: it is also better. Postgres full-text search has no IDF at all — ts_rank_cd scores on term frequency and position, corpus-blind. SQLite's bm25() does weight rare terms. The thing I moved to is closer to the minsearch TF-IDF I was replacing than the "proper" database would have been.

And the original project made it cheap. Its own schema.sql opens with a note that these queries already ran on SQLite before the Postgres migration, and that the SQL had been kept portable. Past me left a door open.

Two things did need real work, not placeholder substitution:

-- Postgres
SELECT DISTINCT ON (author) repo, author, github_url, score, author_total_score
FROM projects WHERE author_total_score IS NOT NULL
ORDER BY author, author_total_score DESC

-- SQLite: no DISTINCT ON. GROUP BY with MAX() returns the row that produced
-- the max, which is exactly the row DISTINCT ON would have kept.
SELECT repo, author, github_url, score, MAX(author_total_score) AS author_total_score
FROM projects WHERE author_total_score IS NOT NULL AND author IS NOT NULL
GROUP BY author ORDER BY author_total_score DESC LIMIT ?

And ARRAY_AGG(repo ORDER BY attempt) has no portable equivalent — SQLite only gained ordered aggregates in 3.44, and the Lambda runtime may ship older. That grouping moved into Python, which is duller and correct.

Decision 2: no streaming, and the reason isn't performance

I had planned a streaming Function URL. Then: Lambda response streaming is supported on Node.js managed runtimes only. Python needs a custom runtime or the Lambda Web Adapter.

I could have fought that. I didn't, because streaming buys less here than it looks. This router runs up to six turns of tool calls, plus a query rewrite, plus a rerank — streaming only helps the final call, so the user waits through the entire tool chain either way.

There's also a billing trap worth naming: a client that disconnects mid-stream still bills the full function duration and the full token spend. On a public demo with a shared budget cap, that's a hole someone can walk through by closing tabs.

Buffered JSON. Moving on.

Decision 3: the auth I designed doesn't work

Plan: S3 + CloudFront with Origin Access Control, Lambda Function URL on AuthType: AWS_IAM, OAC signing requests to the Lambda. Clean, no shared secrets.

It does not work for POST. CloudFront signs the request but does not compute the body payload hash, and Lambda rejects unsigned payloads. /api/ask and /api/feedback are both POSTs. Every one would fail with InvalidSignatureException.

API Gateway solves auth — and reintroduces a ~29-second integration timeout that a six-turn loop can exceed.

So: Function URL on AuthType: NONE, with CloudFront attaching a shared secret header, and the Lambda comparing it with secrets.compare_digest. Which leads to an asymmetry I find genuinely interesting:

• The Lambda reads the secret from Secrets Manager at cold start, because a Lambda environment variable is stored in plaintext in the CloudFormation template.
• CloudFront cannot. Custom origin headers have no runtime secret reference. The value is resolved at deploy time into the distribution config, where anyone with console read access can see it.

That is inherent to the feature, not something I could engineer around. So I stopped pretending it was a credential and wrote down what it actually is: a proof-of-origin token. The rate limiter and the daily budget cap sit behind it precisely because it is not a strong boundary. Both halves are now pinned by tests, so the weakness stays a known property rather than a surprise.

Decision 4: Bedrock batch can't do the thing the whole design rests on

The original enforces a hard rule: only the live chat router makes synchronous LLM calls. Everything bulk — theme classification, ground-truth generation, RAG evaluation, judge scoring, about 11,000 requests — goes through the OpenAI Batch API, at roughly half price.

Two discoveries, in increasing order of annoyance:

1. The Anthropic/OpenAI-style Message Batches API doesn't exist on Bedrock. Fine — Bedrock has its own batch inference: JSONL to S3, CreateModelInvocationJob, poll, JSONL back. Same discount, same shape, different API.

2. Bedrock batch inference supports neither tool calling nor structured output. That one hurts. Four of the five bulk stages relied on text_format=<pydantic model> for guaranteed-schema-valid JSON.

So structure moved from a guarantee into a request:

def json_instruction(schema: dict) -> str:
    """The prompt fragment that stands in for structured output."""
    lines = []
    for name, spec in (schema.get("properties") or {}).items():
        bits = [f'  "{name}": {spec.get("type", "string")}']
        if spec.get("enum"):
            bits.append(f'one of {", ".join(map(str, spec["enum"]))}')
        if spec.get("description"):
            bits.append(spec["description"])
        lines.append(" -- ".join(bits))
    return JSON_INSTRUCTION.format(schema="\n".join(lines))

…and parsing became forgiving on the way back — fenced blocks, preambles, trailing explanations. Not because sloppiness is fine, but because discarding an otherwise-correct answer over a stray sentence means re-running a paid batch job. A record that yields nothing parseable returns None for a retry list rather than raising and killing the run.

The live judge is unaffected: Converse's toolConfig does constrain output. Two code paths, deliberately, and the reason is written at both call sites.

The nice part: I kept the seam in one module. _batch.py retained its function names and request shapes and became a translation layer onto Bedrock, so the prompts, parsing, and stage logic in four consumer modules never changed.

Decision 5: what S3 Vectors should actually store

The obvious design puts chunk text in vector metadata, so one query returns everything the answer needs.

Two constraints argue against it, and both are unforgiving:

• Filterable metadata is capped at 2 KB per vector, and keys are filterable by default. The p90 chunk here is 2.8 KB — leaving content filterable rejects most of the corpus.
• nonFilterableMetadataKeys is fixed at index creation and cannot be changed. Get it wrong and you rebuild.

But the deciding argument was simpler. The SQLite artifact is already open in /tmp. Hydrating text from it is a local file read. Duplicating 31 MB of text into vector metadata means paying to write and store a second copy, and living against the 40 KB-per-vector total cap forever.

S3 Vectors stores identity and filters. SQLite stores text. Vector metadata is now a few hundred bytes.

One correctness trap fell out of splitting retrieval across two systems, and it's the kind that no test of either half would catch: the S3 Vectors metadata filter and the SQLite WHERE clause must accept exactly the same fields. If they diverge, RRF fuses two candidate pools selected under different predicates, and the ranking is quiet wrong. There is now a test whose entire job is asserting those two tuples are equal.

Five decisions were overridden by AWS's terms and conditions: a pricing floor, a runtime limit, a signing gap, the batch API's capabilities, and a metadata limit—rules that conflicted with my original plan.