N.V.J.K KartikKartik

Peer-reviewed paper · IIT Bombay · IEEE TALE 2024

Speaker diarization in classrooms

Best Student Paper

Who said what, in a classroom where people switch between English and local languages mid-sentence?

Plate: Speaker diarization in classrooms.

What I did

Co-author, second of seven. My part was the diarization pipeline work.

I implemented speaker diarization for code-switched classroom audio, with pyannote and Whisper.

IEEE Xplore ↗TALE 2024 awards ↗IIT Bombay ET awards ↗

Authors

Aarsh Desai, N.V.J.K Kartik, Priyesh Gupta, Vinayak, Ashwin T S, Manjunath K. Vanahalli, Ramkumar Rajendran

The call I made

Don’t trust one system with mixed-language speech

Off-the-shelf diarization struggles when a class moves between English and local languages mid-sentence. Combine a diarizer with Whisper’s transcription, a custom voice-activity detector and a clustering step, and measure it against the commercial options.

  1. Classroom audioEnglish and local languages, mid-sentence
  2. Custom voice-activity detection
  3. pyannote diarization + Whisper transcription
  4. Embedding clusteringwho is who
  5. DER 0.26the lowest of the systems compared

Full title

Advancing Speaker Diarization With Whisper Speech Recognition for Different Learning Environments

In plain words

Speaker diarization splits a recording by speaker: this stretch is the teacher, this one is student two. It is the first step before you can study how a class actually talks.

Off-the-shelf systems struggle with Indian classrooms, where speech moves between English and local languages. The paper tests whether a combined pipeline does better.

How it works

  • The pipeline combines pyannote.audio diarization with OpenAI Whisper transcription, a custom voice-activity detector, and an embedding-clustering step.
  • Evaluation used 137 recordings from an online HCI course and an augmented-reality classroom.

What it established

  • The pyannote + Whisper pipeline had the lowest Diarization Error Rate of the systems compared (DER 0.26), ahead of commercial options including Deepgram and Otter.
  • It won the Best Student Paper award at IEEE TALE 2024.
Where it stops
  • A DER of 0.26 still means roughly a quarter of speaking time is attributed to the wrong speaker, or missed. Better than the alternatives is not the same as solved.
  • Two learning settings. How well it transfers to other classrooms and languages is open.

Receipts

  1. IEEE Xplore — Advancing Speaker Diarization With Whisper… (TALE 2024)Method (pyannote + Whisper, custom VAD, embedding clustering), 137 recordings, DER 0.26 vs Deepgram/Otter.
  2. IEEE TALE 2024 — AwardsBest Student Paper; full author order (Aarsh Desai first, Kartik second).
  3. IIT Bombay Educational Technology — AwardsIndependent listing of the TALE 2024 Best Student Paper.
  4. Previous portfolio, as it stood before this redesign (self-reported)Self-reported: IIIT Dharwad (Data Science & AI), research at IIT Bombay Educational Technology, CRF approach for SSMR.

Every source on the site

Next, a little further back in timeMarket sentiment →