I wanted to find out how far I could go in testing a voice agent before it ceased to seem like a demo and began to seem like me. Not something that has a human-sounding voice — I mean an agent that talks, thinks, and arranges things the way I actually do, using my own way of speaking, my own catchphrases, and my own method of hedging a sentence.
It's a lot more difficult than it appears, and the main source of the difficulty is not in the place one would anticipate.
The setup
For the stack, I recorded around 40 minutes of my own voice—comprising scripted statements to ensure coverage of all phonemes and unscripted, loose remarks to achieve a natural speaking rhythm and to include filler words—then fed this material into a voice model that had been finely tuned on Cartesia, with Vapi being used to orchestrate the process and Twilio taking care of the phone functionality. Additionally, I created a prompt pipeline using my resume and project history so that the agent would be able to talk knowledgeably about my background rather than inventing a version of me that doesn't exist.
Putting the 80% together was a matter of wiring the various components together. The agent framework provided by Vapi deals with a lot of the call orchestration right from the start—such as detecting when a call is being made, feeding the speech recognition output continuously into the LLM, and then sending the response back out to Cartesia for synthesis. The framework, however, does not deal with anything that has to do with sounding like a specific person, and it is in this area that the bulk of my actual work consisted.
Where it actually got hard
No one ever tells you this about voice agents, namely that it's not difficult to get the model to be correct. What is difficult instead is achieving the right rhythm in a conversation—knowing when to speak and when to remain silent, and how to deal with an interruption without sounding as if you're operating like a faulty IVR system. Some of the specific problems I encountered were:
- The agent would interrupt by speaking before I had actually finished my sentence. Vapi's default method of detecting when to end a turn (which is based on silence) regards any pause of more than a few hundred milliseconds as a signal that 'your turn has ended' — this works all right in the case of brief exchanges but is不利 for people who pause while thinking. – The cloned voice had a most unnaturally flat tone when giving longer answers. Although Cartesia's model did a good job of reproducing my timbre, it failed to replicate my real cadence, since it read out multi-sentence replies in a monotone that didn't match the way I actually delivered them, particularly when using filler phrases and asides. – The language model always provided excessive explanations by default. When left unimpeded, it responded to every question in the manner of a formal biography rather than as I would when having a casual conversation. It continuously included qualifiers and reservations that I wouldn't actually make when speaking casually.
I decided to set the endpointing thresholds on a per-context basis (increasing the tolerance for longer pauses when generating 'thinking' responses), to divide longer answers into shorter, more conversational segments with natural pauses between them rather than using a single long TTS pass, and I modified the system prompt to clearly advise the use of brief and informal language, including examples of the way I actually speak, rather than giving the generic instruction to 'be conversational'.
// Simplified example of the endpointing override in the Vapi config
{
"responseDelaySeconds": 0.4,
"llmRequestDelaySeconds": 0.1,
"numWordsToInterruptAssistant": 3,
"endpointing": {
"provider": "vapi",
"silenceTimeoutSeconds": 0.8
}
}
The part I'm still not happy with
I am not entirely satisfied with the way interruption handling has been implemented. Although it works fine in a straightforward conversation where only one person speaks at a time, it still sometimes cuts itself off in the middle of a sentence when I make a minor acknowledgent such as 'mm' or 'yeah' while it is speaking — the system interprets any sound as an interruption signal and does not manage to tell the difference between backchannel noises and a real attempt by someone to take over the floor. To fix this properly would require training a lightweight classifier on the input audio rather than depending on raw voice-activity detection, even though I haven't yet built that.
Why this mattered to me
Most demonstrations of 'AI clones' are just gimmicks—theming a voice filter onto a standard chatbot. I wanted to find out whether a truly personalised, real-time voice agent could be built by a single person using off-the-shelf tools (Vapi, Cartesia, Twilio) rather than by a team employing custom TTS infrastructure. The straightforward answer is that mostly yes, but the final 20 per cent—getting it to feel like a person rather than just a feature—accounted for most of the actual work.
The final 20% when you're developing something which involves an AI having a real-time conversation is what makes the difference.
When constructing something similar, focus on the conversational rhythm before you deal with the voice model since it's the element that really decides whether it comes across as alive.
If you're planning to build something similar, please feel free to exchange ideas — you can get in touch with me at deglurkarom@gmail.com or at github.com/deglurkarom-dev.
