Speaker diarization answers ‘which acoustic voice is speaking when?’ rather than ‘who is this person?’ That distinction matters when labels enter legal records, documentary subtitles or automated meeting summaries.
Transcription, alignment and speaker turns
WhisperX combines speech recognition with forced alignment and optional pyannote-based speaker diarization. The pyannote speaker-diarization-3.1 model card describes a 16 kHz mono workflow and PyTorch inference. Treat those as independent stages: transcription can be accurate while the timestamps are wrong; timestamps can align to a misheard phrase; labels can be consistent without identifying anybody. Each stage needs its own acceptance checks.
Access and reproducibility
The pyannote model card requires accepting the relevant model conditions and using a Hugging Face access token. Store the model revision and local dependency versions alongside the resulting annotation, and review the full license of every model and package in the pipeline. A repository being public is not a blanket grant of rights over input recordings or their speakers.
Failure cases to expect
Diarization struggles when two voices overlap, one person whispers then shouts, participants use the same microphone or a voice is heavily processed. Stereo channels sometimes help separate speakers without model inference; a mono conversion can throw away that separation. When you have independent interview tracks, preserve track identity before downmixing. Do not map Speaker 1 to a real name automatically without an independent identity record.
An auditable correction workflow
Split a representative recording into regions, let a reviewer mark speaker-turn start and end times and compute diarization error rate under a declared overlap policy. Compare the model labels to reference speakers and inspect the largest confusion regions. Keep manual corrections as separate edits. The final caption file should use names only after a permitted identity check and should not publish biometric embeddings.
Alternatives
AWS Transcribe and Azure Speech offer managed diarization paths; Deepgram provides explicit versioned diarizers. Pick a route according to region, data controls and the number of speakers, not by a generic speaker recognition label. If only one presenter speaks, diarization may add needless error and processing cost.
Worked production example
Consider a panel with a moderator, two guests and a remote caller. If recording channels are available, preserve them; they often give better direct speaker evidence than an AI guess. Convert a copy to the sampling and channel format required by the chosen aligner, then perform transcription, alignment and diarization separately. On every disputed turn, compare the output with the original track. Overlap should be marked as overlap where it is not possible to assign each word reliably to a sole speaker.
Practical acceptance gate
Build a correction table containing start time, end time, acoustic speaker label, independently verified public name if permitted, review status and editor initials. Ask reviewers to focus on the first turn after silence and on identical-sounding voices, where confusion is common. Do not publish embeddings or claim that a diarization model provides biometric authentication. The model’s terms acceptance, revision and local code licenses should be archived together with each project’s processing manifest.
Region, plan and model-version notes
Documentation checked on 2026-10-25. Exact applicability: availability varies by account/endpoint and jurisdiction. Version: WhisperX / pyannote speaker-diarization-3.1. Plan: not independently verified for a specific plan. Monetary prices and quotas are not asserted unless separately verified; check the canonical AI 10 pricing ledger before purchase or publication of a tariff.
Sources & Further Reading
- Whisperx — official documentation (checked 2026-10-25)
- Pyannote — official documentation (checked 2026-10-25)
- Aws Diar — official documentation (checked 2026-10-25)
- Deep Diar — official documentation (checked 2026-10-25)