You have a talking-head video. The audio and the lips are close, but not exact. Maybe the sound runs three frames ahead. Maybe two people are on screen and you need to know which mouth the voice belongs to. How do you measure that automatically, without a human watching every clip?
That is the problem SyncNet solves, and the open-source Wunjo Make repo ships two versions of it. I want to walk through the actual code: how the network is built, how audio turns into the thing the network eats, and how the codebase turns the network's output into a usable offset and a confidence number.
Plain version first. You build two small networks. One looks at the lips. One listens to the sound. Each one squeezes its input down to a short vector. If the lips and the sound belong together, the two vectors land close. If they do not, the vectors land far apart. Distance is the whole game.
Two towers, never mixed
Here is the first version, SyncNet_color in portable/src/visual_processing/lip_sync/syncnet.py. It is the Wav2Lip-style net. The shape of the very first layer tells you almost everything:
self.face_encoder = nn.Sequential(
Conv2d(15, 32, kernel_size=(7, 7), stride=1, padding=3),
...
)
self.audio_encoder = nn.Sequential(
Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
...
)
The face tower starts at 15 input channels. The audio tower starts at 1. Those two numbers are the design.
Why 15? Because a single still frame cannot tell you anything about motion, and lip-sync is motion. So the face tower does not take one frame. It takes five frames, stacked. Five RGB frames is 5 times 3, which is 15 channels. The network sees a short burst of mouth movement in one shot, not a single freeze-frame.
The audio tower takes 1 channel because a mel spectrogram is a single grayscale image: time on one axis, frequency on the other, loudness as brightness. One channel is all it needs.
The two towers never share weights and never touch inside the network. They run in parallel, on different inputs, and only meet at the very end as two vectors. That is the contrastive idea: learn a face space and a sound space that line up only for matching pairs.
The trick at the bottom: normalize, then compare
The forward pass is short, and the last three lines are the part that matters:
def forward(self, audio_sequences, face_sequences):
face_embedding = self.face_encoder(face_sequences)
audio_embedding = self.audio_encoder(audio_sequences)
audio_embedding = audio_embedding.view(audio_embedding.size(0), -1)
face_embedding = face_embedding.view(face_embedding.size(0), -1)
audio_embedding = F.normalize(audio_embedding, p=2, dim=1)
face_embedding = F.normalize(face_embedding, p=2, dim=1)
return audio_embedding, face_embedding
Both embeddings get L2-normalized. That means every vector is pushed onto a unit sphere: only its direction survives, its length is thrown away. Once both vectors have length one, Euclidean distance and cosine similarity carry the same information. A small distance means the directions agree, which means the lips and the sound match.
This is the part people skip when they reimplement these nets and then wonder why training is unstable. If you compare raw embeddings, a loud frame or a bright frame can pump up the vector length and swamp the signal you actually care about. Normalizing first removes magnitude from the equation. You are left comparing shape, not volume.
The forward pass returns the two vectors. It does not return a score. The caller decides what distance means.
Lining up sound and picture is a matching problem, not a generation problem. Photo: Unsplash
Feeding the audio tower: wav to mel
The audio tower wants a mel window, and portable/src/visual_processing/lip_sync/wav2mel.py builds it. The interesting part is the chunking, where audio time gets cut to match video time:
def chunk_mel(self, mel, fps, mel_step_size):
mel_chunks = []
mel_idx_multiplier = 80./fps
i = 0
while True:
start_idx = int(i * mel_idx_multiplier)
if start_idx + mel_step_size > len(mel[0]):
mel_chunks.append(mel[:, len(mel[0]) - mel_step_size:])
break
mel_chunks.append(mel[:, start_idx : start_idx + mel_step_size])
i += 1
return mel_chunks
mel_step_size is 16. So each chunk is a fixed 16-column slice of the spectrogram, which is the window the audio tower is built to swallow.
The number 80./fps is the bridge between two clocks. The mel is computed at 16000 Hz with a 200-sample hop (see hparams.py), which works out to 80 mel columns per second of audio. Video runs at fps frames per second. So 80./fps is "how many mel columns advance per video frame." At 25 fps that is 3.2 columns per frame. The loop steps one video frame at a time and grabs the matching audio window each time, so chunk number i lines up with video frame number i. The last chunk clamps to the end of the array so you never run off the edge.
There is also a small, honest gotcha baked into the code:
def check_for_nan(self, mel):
if np.isnan(mel.reshape(-1)).sum() > 0:
raise ValueError('Mel contains nan! Using a TTS voice? '
'Add a small epsilon noise to the wav file and try again')
Dead-silent synthetic audio can produce a log of zero, which is a NaN, which poisons the whole spectrogram. The fix in the message is blunt and it works: add a hair of noise so the log has something to bite on.
Where the scoring actually happens
Now the honest part, because the code does not lie about it. SyncNet_color is exported from the module, but at inference time it is not wired up to produce a sync score. The syncnet_wt you see in hparams.py is a training-time loss weight for the lip-sync generator, not a runtime judge. So if you are looking for the place that actually scores audio against video in this repo, it is the other SyncNet.
That second version lives in portable/src/visual_processing/face_detection/sync_net/, and it is the classic Chung and Zisserman SyncNet (ACCV 2016). Same two-tower idea, different build: a 3D-conv face tower and an MFCC audio tower, with separate fully connected heads forward_lip and forward_aud in model.py. This one is alive and doing work.
The work is the offset search, in instance.py:
def calc_pdist(feat1, feat2, vshift=10):
win_size = vshift * 2 + 1
feat2p = torch.nn.functional.pad(feat2, (0, 0, vshift, vshift))
dists = []
for i in range(0, len(feat1)):
dists.append(
torch.nn.functional.pairwise_distance(
feat1[[i], :].repeat(win_size, 1),
feat2p[i:i + win_size, :]))
return dists
Read it as a sliding test. For each video frame's face vector, it does not just compare against the audio vector at the same instant. It compares against a window of audio vectors, from vshift frames early to vshift frames late. That gives you a distance for every possible small shift between the two streams.
Then evaluate collapses that into an answer:
mdist = torch.mean(torch.stack(dists_list, 1), 1)
minval, minidx = torch.min(mdist, 0)
offset = vshift - minidx
conf = torch.median(mdist) - minval
Average the distances across all frames for each shift. The shift with the smallest average distance is the true audio-video offset, because that is where the lips and the sound agree the most. The confidence is the gap between the typical distance and the best distance. A big gap means one shift clearly won, so you can trust it. A tiny gap means every shift looked about the same, so the audio probably is not driving this face at all.
Picking the right face from a crowd
The payoff is in recognition.py, in FaceRecognitionSpeaker. When a video has several faces, the code crops each face track, runs syncnet.evaluate on each one, and keeps the per-frame confidence:
offset, conf, dist = syncnet.evaluate(tmp_dir, video_file=fname)
...
fconf = np.median(mean_dists) - fdist
fconfm = signal.medfilt(fconf, kernel_size=9)
...
if conf > threshold:
if scene_dict is None or scene_dict.get("conf") is None or scene_dict.get("conf") < conf:
scenes_dict[offset_idx] = {"x": x, "y": y, "idx": idx, "conf": conf}
For each frame the code keeps the track with the highest sync confidence, as long as it clears threshold (default 3.0). That is active-speaker detection built entirely on distance: the mouth whose movements match the soundtrack best is the one talking. The medfilt with a kernel of 9 smooths the per-frame confidence so a couple of noisy frames cannot flip the decision.
So the same plain idea drives both versions. Two towers, two vectors, compare by distance. The second version just adds the sliding window so distance gives you an offset and a trust score, not only a yes or no.
 |
About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn. |
Things that will trip you up
A few practical notes if you go poking at this code.
The two SyncNets are not interchangeable. SyncNet_color eats 15-channel stacked RGB and a mel window. SyncNetInstance eats a 5-frame 3D-conv volume and an MFCC slice. Different preprocessing, different checkpoints. Do not feed one's input to the other.
The audio clock is fixed at 16000 Hz and the offset search assumes 25 fps video. The evaluate function normalizes input to 25 fps first for exactly this reason. If your fps drifts, your offset frame count drifts with it.
The confidence is relative, not absolute. It is median distance minus minimum distance, so it tells you how clearly one shift beat the rest for this clip. It is not a percentage you can compare across different videos without care.
If you want to read it all yourself, the model is in portable/src/visual_processing/lip_sync/syncnet.py, the audio prep in wav2mel.py, and the live scoring path in face_detection/sync_net/instance.py and face_detection/recognition.py.
References
- Chung, Zisserman. "Out of Time: Automated Lip Sync in the Wild." ACCV 2016 Workshops. The original two-tower SyncNet and the offset-by-distance idea. https://www.robots.ox.ac.uk/~vgg/publications/2016/Chung16a/
- Prajwal, Mukhopadhyay, Namboodiri, Jawahar. "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild." ACM Multimedia 2020. arXiv:2008.10010. The Wav2Lip line that
SyncNet_colorbelongs to. https://arxiv.org/abs/2008.10010
