The field is young enough that nobody has ten years of experience, so interviewers weight engineering judgement heavily over tool familiarity. The questions are less about what you have used and more about how you would know it was working.
Almost every candidate has now built something with a language model. That means the demo no longer differentiates anyone, and interviews have moved to the parts that are hard: making retrieval actually relevant, knowing whether a change made things worse, and handling text you did not write.
Retrieval, past the diagram
Everyone can draw the pipeline: embed the documents, embed the question, find the nearest, put them in the context. Interviewers have seen it. The questions are all about what makes it work badly.
- Chunking. Too small and a chunk loses the context that made it meaningful. Too large and the relevant sentence is diluted by everything around it. There is no universal size, and saying so with a reason is better than naming one.
- Semantic search missing exact matches. Nearest-neighbour search is poor at identifiers, error codes and product names, which is why hybrid retrieval with a keyword component so often beats pure embeddings in practice.
- Retrieving the wrong thing confidently. Similarity is not relevance. Something always comes back, whether or not anything useful exists.
- Stale content. The index is a copy. What happens when the source changes is an operational question people forget to answer.
Users say the answers are often wrong. How would you debug that?
Junior answer: I would improve the instructions to the model, try a larger model, and adjust the temperature to make it more accurate and less prone to making things up.
Senior answer: First I would find out whether retrieval or generation is failing, because they need opposite fixes and guessing wastes a week. I would take a sample of the bad answers and look at what was actually retrieved. If the right passage was never retrieved, no amount of instruction tuning will help and the work is in chunking or hybrid search. If the right passage was retrieved and the answer still contradicted it, that is a generation and grounding problem. Before changing anything I would build a small set of question and expected-answer pairs, so I can tell whether the next change helped or just moved the failures somewhere I was not looking.
Splitting retrieval failure from generation failure is the single most useful diagnostic in this area, and most candidates skip straight to the model.
Evaluation, which is the real interview
If there is one topic that separates people who have shipped this from people who have prototyped it, this is it. Non-deterministic output means you cannot rely on the usual assertion-based safety net, and the temptation is to test by trying a few things and feeling good about them.
What a strong answer describes is a fixed evaluation set built from real failures, run on every change, with results compared rather than eyeballed. Then the honest caveats: automated judging is itself unreliable and needs spot-checking, and an evaluation set drifts away from real usage unless you keep feeding production failures back into it.
Say that you would add every reported bad answer to the set as a regression case and you have described the practice that actually keeps these systems from decaying.
Getting structured output reliably
A practical question because it is what most production integrations need. The model produces text; your code needs a shape it can rely on.
The expected answer has three parts: constrain the output where the platform supports it, validate against a schema on the way in rather than trusting it, and have a defined path when validation fails. That last part is where the follow-up goes. Retrying forever is a cost and rate-limit incident waiting to happen, so a bounded retry and then a fallback is the answer that shows you have run this in production.
The framing that lands
Treat model output as untrusted input, exactly like a request body from the internet. Validate it, bound it, and never let it reach anything consequential unchecked. That single sentence answers a surprising number of questions in this area.
Untrusted text and injection
Increasingly asked, and the answers separate people quickly.
The core problem is that instructions and data arrive through the same channel, so text from a document, a web page or a user message can contain something that reads as an instruction. There is no complete fix through wording alone, and a candidate who claims otherwise has not thought about it hard.
What actually reduces risk is architectural: keep untrusted content clearly separated from your own instructions, do not grant the system capabilities it does not need, require confirmation before anything consequential or irreversible, and validate any tool call before executing it. The mindset here is the same one a security round tests, applied to a component that is non-deterministic.
Cost and latency
Frequently the most practical section, and the one where product-minded engineers do well.
Cost scales with tokens, so long context is a per-request bill, not a one-off design choice. Latency scales with output length in particular, which is why streaming exists and why it changes perceived speed far more than raw model speed. The senior framing is routing: not every request needs the most capable option, and using a cheaper path for the easy majority is usually the largest available win.
Fine-tuning versus retrieval versus better instructions
A judgement question, and the wrong answer is reaching for fine-tuning first.
The distinction to draw is between knowledge and behaviour. If the system needs facts it does not have, that is retrieval, and fine-tuning is an expensive way to bake in something that will be stale next month. If it needs a consistent format or style, that is a case for tuning, once you have exhausted cheaper options. Being explicit that you would start with the cheapest reversible change is the signal.
Common questions
Do I need to know specific frameworks?
Much less than you would think. The field moves fast enough that interviewers weight reasoning about retrieval quality, evaluation and failure handling well above tool familiarity.
Is machine learning theory expected?
For an AI engineering role, usually not. Embeddings and why similarity is not relevance, yes. Gradient descent, rarely, unless the role is actually a modelling role.
How do I show experience if my project was small?
Talk about what went wrong and what you measured. A small project with a real evaluation set and a documented failure mode is stronger evidence than a large one with neither.
This area is almost entirely judgement under follow-up, which is exactly what a focused practice round exposes. Practise an AI engineering interview
