AudioSeg

AudioSeg predicts topic boundaries directly from audio, on a 6-second grid, and was introduced in our paper Beyond Transcripts: A Renewed Perspective on Audio Chaptering (acl | arXiv). Given an audio file (e.g. a podcast, lecture, or YouTube video), the model outputs a boundary probability for every 6-second segment, which can be thresholded into chapter/topic boundaries.

Architecture

• Encoder (frozen): openai/whisper-large-v3 encoder, applied to consecutive 30-second chunks of the audio.
• Head (trained, ~32M params):
  • a local segment transformer that pools encoder frames into one embedding per 6-second segment via a learned [SEG] token
  • a document encoder over the segment sequence producing per-segment boundary probabilities

Only the head weights are stored in this repository; the Whisper encoder is downloaded from openai/whisper-large-v3 on first use.

Usage

pip install transformers torch torchaudio

from transformers import AutoModel

model = AutoModel.from_pretrained("retkowski/audioseg", trust_remote_code=True)
model = model.to("cuda")

result = model.segment("episode.mp3")

result["ts_boundaries"]     # boundary timestamps in seconds, e.g. [315.0, 747.0, ...]
result["probs"]             # boundary probability per 6s segment
result["segment_size_sec"]  # 6.0

segment() also accepts a waveform tensor (model.segment(waveform, sample_rate=sr)), a custom decision threshold (default 0.5), and batch_chunks to control how many 30-second chunks are encoded per Whisper batch.

A reported timestamp t is the midpoint of the 6-second segment in which a topic change was detected, i.e. the change lies within [t - 3, t + 3) seconds. The start of the audio is not considered a boundary, so the first segment is never reported.

Training

The model was trained on YTSeg, YouTube videos with human-created chapter markers, using the chapter start times as boundary supervision (binary cross-entropy on the 6-second grid). The Whisper encoder was kept frozen.

Citing

We kindly request you to cite our corresponding paper if you use our model:

@inproceedings{retkowski-etal-2026-beyond,
    title = "Beyond Transcripts: A Renewed Perspective on Audio Chaptering",
    author = {Retkowski, Fabian  and
      Z{\"u}fle, Maike  and
      Nguyen, Thai Binh  and
      Niehues, Jan  and
      Waibel, Alexander},
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.396/",
    doi = "10.18653/v1/2026.acl-long.396",
    pages = "8765--8787",
    ISBN = "979-8-89176-390-6",
    abstract = "Audio chaptering, the task of automatically segmenting long-form audio into coherent sections, is increasingly important for navigating podcasts, lectures, and videos. Despite its relevance, research remains limited and text-based, leaving key questions unresolved about leveraging audio information, handling ASR errors, and transcript-free evaluation. We address these gaps through three contributions: (1) a systematic comparison between text-based models with acoustic features, a novel audio-only architecture (AudioSeg) operating on learned audio representations, and multimodal LLMs; (2) empirical analysis of factors affecting performance, including transcript quality, acoustic features, duration, and speaker composition; and (3) formalized evaluation protocols contrasting transcript-dependent text-space protocols with transcript-invariant time-space protocols. Our experiments on YTSeg reveal that AudioSeg substantially outperforms text-based approaches, pauses provide the largest acoustic gains, and current MLLMs struggle due to context limitations and weak instruction following."
}

License

CC-BY-NC-4.0 (non-commercial use).