Deepfake Voice Detection for Call Centers [2026]: Deploy It Right

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

Deepfake voice detection for call centers is what you deploy when you’re done arguing about whether voice cloning is “real” and you’re ready to stop getting social-engineered on live calls.

It’s a set of real-time controls that decide whether a caller’s audio is synthetic enough to require step-up verification, without breaking the call flow or humiliating legitimate customers. Generic “audio deepfake detection” assumes clean WAV files and offline scoring. Call centers get narrowband VoIP, echo cancellation, packet loss concealment, and a hard constraint: if you annoy real customers, you’ve built a fraud system that creates fraud.

Key takeaways

• Treat deepfake detection like an SLO’d production system, not a model demo. Your real enemy is latency, codec mismatch, and operational handling.
• The cleanest tap point is usually your Session Border Controller (SBC) or media relay, pre-mix and pre-AEC if you can get it. Everything downstream gets uglier fast.
• Use a 3-tier action policy (low/medium/high confidence) with step-up verification, not auto-blocks. False positives are a CX incident.
• Don’t ship without an appeals workflow and audit artifacts. Vendor blog URLs rot; your compliance evidence can’t.
• Combine audio detection with caller authentication and fraud signals. STIR/SHAKEN helps with caller ID spoofing, but it doesn’t validate the voice content.

If your deepfake detector can’t explain what happens after a flag, you don’t have a detector. You have a liability.

Why call-center deepfake detection is different (and harder than it looks)

The first mistake teams make is treating this like image deepfakes: run a classifier, set a threshold, done.

That mental model is wrong for call centers.

In a contact center, you’re not trying to win a forensic argument about whether the waveform is “fake.” You’re trying to prevent account takeover and social engineering while keeping conversion, containment, and handle time stable. So you’re optimizing a messier objective function than “accuracy”:

• False negatives cost money (fraud loss, chargebacks, regulatory exposure).
• False positives cost trust (legitimate customers escalated, blocked, or made to feel like criminals).
• Latency is a first-class constraint (the agent needs guidance while the conversation is still happening).

And you’re doing it on compromised audio. Most call center audio is:

• 8 kHz narrowband somewhere along the chain (PSTN bridges still exist, and plenty of CCaaS pipelines downsample).
• Compressed with codecs like G.711 (often 64 kbps PCM µ-law/A-law) or Opus (variable bitrate, usually tuned for speech).
• Mutated by DSP: automatic gain control (AGC), acoustic echo cancellation (AEC), noise suppression, and sometimes “HD voice” processing that looks great in marketing decks and weird in model features.

A detector trained on studio speech will look brilliant in a demo and then faceplant the week you put it behind an SBC. I’ve watched too many teams learn this the expensive way.

Where should detection run: edge, CCaaS, or fraud backend?

There are three sane deployment locations. Pick based on latency budget, privacy boundary, and audio cleanliness. Everything else is bikeshedding.

Option A: Agent device / “edge” (softphone or desktop)

It’s tempting because it’s close to the agent UX. You can throw warnings on screen instantly.

I’m skeptical of this approach unless you’re already a desktop platform company.

• Attack surface: you’re shipping a security control to endpoints you don’t fully control.
• Operational drift: versions and configs diverge across thousands of agents. You will be debugging “why did it flag on this laptop but not that one?” at 2 a.m.
• Audio quality: you often only see post-processed audio after the client DSP stack has chewed it up.

Use this when:

• You must keep audio local for policy reasons.
• You can ship a managed desktop client and you actually have update control.

Option B: Inside your CCaaS / telephony fabric (SBC/media relay)

For most teams, this is the sweet spot.

• You can access RTP streams in a consistent place.
• You can run streaming inference with predictable compute.
• You can attach metadata to the call in real time.

The practical implementation pattern is usually: fork RTP or use standardized recording flows like SIPREC.

SIPREC (Session Recording Protocol) is an IETF standard that specifies using SIP/SDP/RTP to deliver real-time media and metadata to a recording device. The abstract in Lyle Portman’s RFC spells out the core model: an on-path Session Recording Client (SRC) streams media to a Session Recording Server (SRS).

Even if you don’t use SIPREC verbatim, the mental model matters: something on-path is responsible for getting you audio you can trust.

Use this when:

• You need sub-second actions (agent prompts, step-up scripts).
• You want the cleanest audio tap point you can get.

Option C: Fraud backend (post-call or near-real-time)

This is the easiest to build and the easiest to justify organizationally. You don’t touch telephony. You consume recordings.

The catch is obvious: you lose the ability to intervene mid-call. At best, you can:

• Flag accounts for post-call review
• Trigger downstream controls (password reset, transaction hold, limits)

Use this when:

• Your priority is investigation and analytics, not real-time interruption.
• Your call flows are too fragile (or political) to risk inserting real-time friction.

Edge vs server vs CCaaS: comparison table

Here’s the comparison I use when advising teams. It’s not “fair.” It’s just reality.

If you can only build one thing this quarter, build the CCaaS/SBC integration. Everything else is a compromise you’ll pay for later.

What audio signals survive VoIP codecs (and what breaks)

Deepfake detectors learn artifacts. VoIP also introduces artifacts. So if you’re careless, your model learns “is this Opus?” instead of “is this synthetic speech?”

ASVspoof is the most widely referenced evaluation series for spoofed/deepfake speech countermeasures. The organizers push common evaluation plans and datasets precisely because vendor claims don’t transfer to your channel. Start here: ASVspoof.

In the Junichi Yamagishi ASVspoof 2021 paper, the authors explicitly call out that channel and compression variability compound difficulty, and they structured tasks to reflect changing conditions. That framing is the point: synthetic speech is adversarial and moving, not a solved classification exercise.

Signals that usually survive (enough to be useful)

On narrowband 8 kHz audio, you can still extract useful cues like:

• Temporal consistency features: micro-timing patterns, weird uniformity in phoneme transitions.
• Prosodic features: rhythm, stress, pitch contours. Narrowband limits detail, but timing survives.
• Some phase-related cues depending on your pipeline (though DSP can obliterate them).

Practical takeaway: prefer detectors designed for telephony bandwidth. If a vendor is selling you a model that “uses ultrasonic artifacts above 8 kHz,” you already know how that ends in a call center. It ends with false confidence.

Signals that break (or become untrustworthy)

In production call audio, these often turn into junk:

• High-frequency spectral artifacts that show up in 16–48 kHz recordings. Narrowband drops them.
• Room acoustics cues after AEC and noise suppression do their thing.
• “Fingerprinting” of synthesis vocoders that disappears after codec + packet loss concealment.

Packet loss concealment can smear or even synthesize small segments of audio. That means your model can accidentally learn PLC artifacts as “deepfake artifacts.” You’ll proudly flag “spoofing” when the real culprit is bad Wi‑Fi.

The “double compression” failure mode

The most underrated production problem is boring: call audio gets transcoded multiple times.

A common chain looks like:

• Caller on mobile network → carrier transcode → PSTN bridge → CCaaS transcode to Opus → recording transcode to MP3

If your model was trained on clean Opus but you score MP3 recordings, performance falls off a cliff. Not because the model is “bad.” Because you changed the physics and pretended you didn’t.

How to ingest audio in production (SIPREC, RTP fork, media relay)

If you can’t get a stable stream, you can’t do real-time anything. This is where most “vendor deepfake detection” pitches go to die.

The tap points that matter

There are four common places to grab audio, ranked best to worst for detection:

1. Pre-mix, per-leg RTP (best): separate caller and agent legs before they’re mixed into a single mono stream.
2. Post-mix RTP: still real time, but you lose channel separation.
3. SIPREC feed: structured recording streams from an on-path SRC to an SRS.
4. Post-call recording (worst): whatever codec and processing the recorder decided you deserve.

If you can get per-leg audio, do it. Separation reduces false positives because the detector can focus on the caller and ignore the agent’s headset noise, keyboard clicks, and office acoustics.

RTP is simple. Your environment isn’t.

RTP itself is a standard transport protocol. Van Jacobson and coauthors describe RTP’s intent clearly: it provides end-to-end transport suitable for real-time data, but it does not guarantee QoS.

That single sentence is your deployment reality: jitter buffers, clock drift, packet loss, retransmits (if you’ve layered them in), and time alignment across legs are now your problem.

Practical architecture: “fork, score, annotate”

A pattern that works in real stacks:

• Fork RTP at the SBC/media relay to a “detection media relay.”
• The relay normalizes: resample to 8 kHz (if narrowband), loudness normalization, optional VAD.
• Stream fixed-size windows to the model server.
• Emit per-window scores + metadata back into your fraud scoring system and agent desktop.

You don’t need to store raw audio to do this. You do need consistent timestamps and session IDs. Sloppy identifiers will wreck your audits later.

Latency budgets: what’s realistic for real-time intervention?

Most teams underestimate how fast decisions need to be. By a lot.

If your goal is an agent warning that actually changes behavior (“don’t reset password yet”), your signal needs to arrive before the agent reaches the risky step in the script.

In practice, I design around two latency tiers:

• Soft intervention: 150–300 ms “heads-up” alerts that can land while the customer is mid-sentence.
• Hard intervention: 500–1500 ms for step-up verification prompts, where you can afford to wait for more evidence.

The windowing tradeoff is brutal:

• Short windows (e.g., 0.5–1.0 seconds) react quickly but are noisy.
• Longer windows (e.g., 2–4 seconds) stabilize but delay action.

A production trick is to run both:

• A fast, cheap model that produces a “suspicion signal” every 500 ms.
• A slower, stronger model that confirms on a 2–4 second rolling window.

If you’ve read my LLM performance writing, this should feel familiar: latency is a UX feature. I’ve written about streaming metrics and what users actually perceive in LLM latency benchmark methodology and how teams hit budgets in LLM latency benchmarks 2026. Voice detection is the same game, just with a different payload.

Thresholds and false positives: a safe playbook (3-tier actions)

A detector score isn’t a verdict. It’s a risk input.

If you want the policy that’s least likely to torch your CX while still reducing fraud, it looks like this.

Tier 1: Low confidence (log only)

• Add a fraud feature: “possible synthetic speech, low confidence.”
• Don’t change the call flow.
• Use it to build your evaluation set.

Tier 2: Medium confidence (step-up verification)

• Trigger a scripted step-up:
  • one-time passcode to an already-enrolled device
  • app push approval
  • out-of-band call-back to a known number
• If the customer fails step-up, don’t accuse them. Just refuse high-risk actions.

This is the security equivalent of “HITL approvals.” The mechanics map cleanly from governance patterns like HITL tool approval patterns.

Tier 3: High confidence (containment + supervisor)

• Freeze the most sensitive actions (password reset, payout changes, address changes).
• Route to a specialist queue or supervisor.
• Keep the customer talking while you perform backend verification.

The key is that Tier 3 is not “hang up.” Hanging up trains attackers and punishes legitimate edge cases (accessibility devices, unusual mics, neurodivergent speech patterns).

The agent script matters more than the model

You need agent language that doesn’t create conflict.

Good script:

• “For security, I need to do an extra verification step before I can change your account details.”

Bad script:

• “Our system thinks your voice is fake.”

That difference is the difference between a resolved call and a viral customer rant.

Appeals and review: the part everyone skips (and regulators will ask for)

If you deploy voice-based flags, you’ve created a system that can impact access to services. You need an appeals path. Full stop.

Here’s what a real appeals workflow looks like in a call center environment.

What to store (without hoarding raw audio)

Store an “evidence packet” per flagged session that includes:

• Call session identifiers (SIP Call-ID / internal session ID)
• Timestamped per-window scores (e.g., every 500 ms)
• Codec and transport metadata (payload type, sampling rate, packet loss stats)
• Tap-point metadata (pre/post mix, pre/post AEC)
• Model version, feature pipeline version, threshold config version
• The action taken (step-up invoked, supervisor routing, action blocked)

You can often avoid storing raw audio entirely. If you must store something, store very short snippets (e.g., 2–4 seconds) with strict retention.

Retention periods and access control

Your retention policy should be explicit:

• Tier 2 flags: retain evidence packet for 30–90 days (enough for QA and dispute).
• Tier 3 flags: retain evidence packet for 180+ days if tied to fraud investigation.

Keep access restricted to fraud ops + QA. Not every supervisor needs to see these details.

Why “stable references” matter (freshness angle)

This sounds petty until you live it: vendor references rot.

A URL that used to be widely cited for voice biometrics and deepfakes now redirects to generic marketing pages (Nuance content shifting under Microsoft branding). When your compliance team asks “why did we block this customer,” “a vendor blog post from 2023” is not evidence.

Your evidence is your own logs, your own configs, and archived standards and benchmarks.

Evaluating and monitoring on real call traffic (without storing sensitive audio)

Lab accuracy is not production accuracy. Always.

If you want one public reality check, follow ASVspoof because it forces comparability across systems. The ASVspoof 2021 paper’s point that matched training/dev data wasn’t provided is basically a statement of the real world: attacks change and you won’t be trained on them.

Build a channel-matched test set

Minimum viable evaluation set:

• 1,000 calls of normal traffic (sampled across geos, devices, languages)
• 200 known fraud calls (however you label them today)
• 200 synthetic attacks generated through your actual telephony chain

That last part matters. Don’t generate pristine audio and score it offline. Generate synthetic speech and run it through:

• your codec chain (G.711/Opus settings)
• your jitter buffer behavior
• your AEC/noise suppression

Red-team with your own codecs

You don’t need a fancy red team. You need a checklist:

• Replay attacks (play a recording into the mic)
• Voice conversion (convert a real voice to a target voice)
• Real-time cloning (human-in-the-loop, low latency)
• Adversarial noise overlays (music, keyboard, office noise)

Track performance over time. The metric that matters is not “accuracy.” It’s something like:

• Tier 2 step-up rate per 10,000 calls
• Tier 3 containment rate per 10,000 calls
• Confirmed fraud prevented per 10,000 calls
• Customer complaints attributable to verification friction

This is the same mental model I use for production systems: if you can’t observe it, you can’t trust it. If you’re building broader controls, my AI security pillar post goes deeper on the governance mechanics.

Defense-in-depth: STIR/SHAKEN, biometrics, KBA, and fraud scoring

Audio deepfake detection should not be your only line of defense. It should be one feature in a risk engine.

STIR/SHAKEN is necessary, not sufficient

STIR/SHAKEN is designed to authenticate caller ID information in VoIP/SIP calls. It helps reduce spoofing, but it doesn’t validate the human voice content itself. That’s why it pairs well with audio-level detection.

If you need a refresher, the STIR/SHAKEN overview explains the protocol family and its intent.

Voice biometrics: useful, but deepfakes target it directly

Voice biometrics can still help for low-risk flows, but attackers are explicitly trying to defeat it with cloning. If you use voice biometrics:

• treat a “match” as one signal, not a login
• combine with device and behavioral signals

Knowledge-based authentication (KBA) is dying

KBA (mother’s maiden name, last 4 digits, etc.) is compromised at internet scale. Deepfake voice makes it worse because social engineering gets more convincing.

Behavioral + device signals are your friend

The highest ROI signals in call center fraud often aren’t audio:

• device fingerprinting in authenticated app flows
• historical contact patterns
• velocity checks (address change + payout change same call)
• agent-side anomalies (unusual script deviations)

The point: your detector should produce a score that feeds a broader fraud decision.

Attacker adaptations you should plan for (and mitigations that work)

Attackers will adapt faster than your model refresh cadence. Plan accordingly.

Replay attacks

Mitigation:

• challenge-response prompts that require real-time cognition
• detect room acoustics inconsistencies (careful: VoIP breaks this)

Voice conversion and “style transfer”

Mitigation:

• look for artifacts in transitions, not steady-state timbre
• fuse with non-audio fraud signals

Human-in-the-loop real-time cloning

This is the scary one: the attacker uses a model that speaks with low latency while a human steers content.

Mitigation:

• don’t rely on “odd wording” heuristics (humans can fix wording)
• rely on step-up verification tied to an enrolled device

Adversarial noise

Mitigation:

• robust preprocessing and VAD
• thresholds that degrade gracefully (don’t spike false positives)

If you’re coming from the LLM world, the analogy is prompt injection: the system is under adversarial pressure. I write about that threat model in prompt injection and agent attack surfaces. Different input. Same principle. Assume the input is hostile.

A deployment checklist you can actually run this quarter

Here’s a sequence you can run without pretending you have infinite time, infinite budget, or infinite patience.

1. Pick the tap point. Start with SBC/media relay RTP fork. Only fall back to recordings if you must.
2. Normalize the stream. Decide your canonical format (often 8 kHz mono PCM internally) and keep it consistent.
3. Select two models or two modes. Fast suspicion + slower confirmation.
4. Define the 3-tier action policy. Log-only, step-up, containment.
5. Ship agent UX and scripts. If agents hate it, your project is dead.
6. Create evidence packets + retention. Make compliance happy before they come looking.
7. Run a 2-week shadow mode. Measure step-up rate and false positive pain.
8. Red-team with your own codecs. Treat transcoding as an attack surface.
9. Roll out gradually by queue. Start with the highest fraud queues.

If you want a mental model for production gates, I learned this the hard way while building this site’s multi-agent publishing pipeline: deterministic quality gates catch more issues than simply “using a bigger model.” Same idea here. You need deterministic operational gates (tap point, codec normalization, threshold policies, evidence logging), not just “a better detector.”

The uncomfortable conclusion: the model is not the product

Deepfake voice detection for call centers is not a classifier you buy. It’s a system you operate.

In 2026, the teams that win won’t be the ones with the fanciest neural net. They’ll be the ones who can hit a 200–300 ms decision budget, keep false positives from turning into CX incidents, and produce an audit packet that still makes sense six months later.

My prediction: within 12–18 months, “deepfake detection” becomes a commodity feature in CCaaS platforms. Your differentiation moves up the stack. The operational layer. The policy. The appeals workflow. The integration with fraud scoring.

If you’re building this now, don’t start by asking “which model is best?” Start by asking: where do I tap RTP, what do I do when I’m wrong, and how fast does the agent need the answer?

Originally published on kunalganglani.com