Abstract
Marimba uses frozen Qwen3-ASR features and lightweight readouts to make lexical, turn-end, language, speaker-change, and personal-information decisions from speech, achieving strong detection and calibration on conversational and read-speech benchmarks.
Speech applications require decisions about term presence, turn completion, and language within an audio window. We present Marimba, an interface for typed speech decisions built on frozen Qwen3-ASR 0.6B features. Lexical questions use zero-shot scoring of ASR transcripts; non-lexical questions use a question-conditioned head and a separate layer-18 turn-end readout. Language identification uses the ASR model's own language scores. On private conversational test sets, decoder AUROC is 0.80-0.92; on public read speech it is 0.92-0.97. Speech Commands four-way accuracy is 0.93; binary lexical ECE is at most 0.08. Personal-information detection, which passes the greedy transcript to a private, curated PII pipeline, reaches item-level F1 0.904 on synthetic Brazilian Portuguese speech (1.000 on reference transcripts). Layer-12 speaker readouts detect speaker change with AUROC 0.966-0.980 on AMI and VoxConverse, above an ECAPA baseline, and count speakers (0, 1, 2, or 3+) more accurately than pyannote 3.1. The turn-end readout reaches AUROC 0.930 versus Smart Turn v3.2 at 0.982 on Smart Turn's own test set. On a subset of ICSI meetings labeled by next speaker, AUROC is 0.716 versus 0.712; the paired confidence interval includes zero.
Community
Marimba is a System One model for speech: ask typed questions about live audio, get typed answers back every 250 ms.
Is someone speaking? Did the turn end? Which language? Did they say "refund"? Did the speaker change? How many speakers?
Answers come back as yes/no, a choice, or a score.
- One frozen Qwen3-ASR-0.6B serves every question, so it runs on a laptop CPU.
- Counts speakers better than pyannote 3.1 on AMI and VoxConverse; 100% language ID on 300 FLEURS clips.
- Open weights, benchmark and code (Apache-2.0).
Latency (p50 per question, 8 s window, batch 1):
| Question | Mac M4 Pro CPU | NVIDIA L4 GPU |
|---|---|---|
| Turn end | ~100 ms | ~12 ms |
| Voice activity | ~105 ms | ~14 ms |
| Language | ~1.3 s | ~150 ms |
| Term ("did they say X?") | not measured | ~1.4 s |
Turn end and voice activity are fast on CPU; term detection needs ASR decoding, so a GPU helps there. Protocol: https://github.com/Charcot-AI/marimba-api/blob/main/docs/benchmarks.md
Try it live: https://huggingface.co/spaces/charcot-ai/marimba
Model: https://huggingface.co/charcot-ai/marimba
Benchmark: https://huggingface.co/datasets/charcot-ai/marimba-bench
Code: https://github.com/Charcot-AI/marimba-api
Author here, ask me anything!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Audio System One Model: Typed Probabilistic Perception of Emotion in Speech (2026)
- Qwen-Audio-Jev: A Unified Interface for Audio Decisions (2026)
- Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen LLM Hear Beyond the Transcript (2026)
- SpeakerTurn: Target-Speaker Semantic Turn Detection in Multi-Speaker Environments (2026)
- Omni-JEV: Text and Dual-Stream Speech Decisions (2026)
- VoxTurn: Answering Several Turn-Taking Questions in One Pass with One Model (2026)
- Walrus Technical Report (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.18330 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 1
charcot-ai/marimba-bench
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper