ModelsLab

/

omnivoice-singing

like 35

Follow

ModelsLab 71

Apologies for the AI post but after trying to get this working for ~2hrs I'm feeling lazy and burnt out, so I asked my cursor agent to just spit out a bug report since it will be more comprehensive (for better or worse) than I'm capable of right now

Model: ModelsLab/omnivoice-singing
Library: omnivoice==0.2.1 (from omnivoice.models.omnivoice import OmniVoice)
Hardware: Windows, RTX 2080 (cuda:1), torch.float16 (Turing; no native bf16)

Summary

The model card says original OmniVoice capabilities including voice cloning are preserved, and emotion tags ([happy], [angry], etc.) are supported. Using the suggested Python API inference path, emotion tags + voice clone produce mangled speech with missing words. The same emotion-tagged line without clone is fine. Plain text + clone (no emotion tag) is fine.

Expected

With ref_audio / ref_text, the model should speak the full target line in the cloned voice, with the emotion tag affecting style only (not deleting or rearranging words).

audios = model.generate(
    text="[happy] Hello adventurer, welcome to Whiterun. The guards have been talking about you.",
    language="English",
    ref_audio="ref.wav",
    ref_text="<transcript of ref.wav>",
)

Should include the full sentence, e.g. “Hello adventurer, welcome to Whiterun. The guards have been talking about you.”

Actual

Listened to outputs from a single controlled run (same ~5s reference clip + matching transcript for all clone cases):

So cloning works without emotion tags, and emotion tags work without cloning — the failure is specifically emotion tag + clone.

Environment

OS: Windows 10
GPU: NVIDIA GeForce RTX 2080
Python: 3.x + venv
pip: omnivoice==0.2.1
torch: 2.9.x + CUDA (cu128)
transformers: 5.x

Minimal reproduction

import soundfile as sf
import torch
from omnivoice.models.omnivoice import OmniVoice

DEVICE = "cuda:0"  # or cuda:1
dtype = torch.float16  # use bfloat16 on Ampere+

model = OmniVoice.from_pretrained(
    "ModelsLab/omnivoice-singing",
    device_map=DEVICE,
    torch_dtype=dtype,
    load_asr=True,
    asr_model_name="openai/whisper-base",
).eval()

# Use a clean ~3–10s mono clip + its true transcript
ref_audio = "ref_5s.wav"
ref_text = model.transcribe(ref_audio)  # or paste a known-correct transcript

text = (
    "[happy] Hello adventurer, welcome to Whiterun. "
    "The guards have been talking about you."
)

# Control: emotion, no clone — fine
audios = model.generate(text=text, language="English")
sf.write("happy_no_clone.wav", audios[0], model.sampling_rate)

# Bug case: emotion + clone
audios = model.generate(
    text=text,
    language="English",
    ref_audio=ref_audio,
    ref_text=ref_text,
)
sf.write("happy_clone.wav", audios[0], model.sampling_rate)

# Compare: same clone, no emotion tag — fine
audios = model.generate(
    text=text.removeprefix("[happy] ").lstrip(),
    language="English",
    ref_audio=ref_audio,
    ref_text=ref_text,
)
sf.write("plain_clone.wav", audios[0], model.sampling_rate)

# Compare: another emotion tag + clone — also mangled
audios = model.generate(
    text="[angry] Get out of my tavern before I throw you into the street myself!",
    language="English",
    ref_audio=ref_audio,
    ref_text=ref_text,
)
sf.write("angry_clone.wav", audios[0], model.sampling_rate)

Notes

• Same architecture/tokenizer files as k2-fsa/OmniVoice; difference is the singing/emotion fine-tune weights.
• Reproduced with both ref_audio+ref_text and create_voice_clone_prompt(...).
• Ref clip was ~5s (within the usual 3–10s clone guidance), with a Whisper transcript of that same clip.

Ask

Is emotion-tag + zero-shot clone a known limitation of this checkpoint, or should clone + leading emotion tags ([happy], [angry], etc.) produce the full target text with the intended style? If there’s a recommended prompt/format for emotion+clone, documenting it on the model card would help a lot.