LFG-2

LFG-2-silver

LFG-2 is a speech-in / text-out voice assistant: a trained audio projector bridging a Gemma E4B audio encoder to a Gemma 4 31B language model. The audio encoder and the language model are frozen; only the projector is trained, so the model inherits the full text-side reasoning of Gemma 4 31B while learning to listen.

  • Input: spoken English audio (16 kHz mono).
  • Output: text. The model reasons in a hidden thought channel (emitting a <heard>โ€ฆ</heard> transcription of what it heard as a comprehension check), then produces the final answer.

Usage

pip install -U transformers accelerate librosa

# On a bare machine, install the torch trio together first (matching CUDA build):
#   pip install torch torchvision torchaudio

LFG-2 loads exactly like base Gemma 4 E4B audio โ€” AutoProcessor + AutoModelForMultimodalLM โ€” the only difference is trust_remote_code=True, which lets the repo's bundled model class install the trained projector for you.

import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM

REPO = "glenn2/LFG-2"
processor = AutoProcessor.from_pretrained(REPO)
model = AutoModelForMultimodalLM.from_pretrained(
    REPO, dtype="auto", device_map="auto", trust_remote_code=True,
)

# Ask a question with your voice (audio can be a local path or a URL; 16 kHz mono).
messages = [
    {"role": "system", "content": "You are a helpful voice assistant. Answer the user's spoken question clearly and concisely."},
    {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]},
]
inputs = processor.apply_chat_template(
    messages, tokenize=True, return_dict=True, return_tensors="pt",
    add_generation_prompt=True, enable_thinking=True,
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=2048)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)

# parse_response strips the <|channel>thoughtโ€ฆ<channel|> block; keep the answer.
print(processor.parse_response(response)["content"])
Without trust_remote_code (manual projector install)

If you'd rather not run repo code, load the base model normally and install the projector yourself (needs huggingface_hub):

import sys, torch
from huggingface_hub import snapshot_download
from transformers import AutoProcessor, AutoModelForMultimodalLM

local = snapshot_download("glenn2/LFG-2")
processor = AutoProcessor.from_pretrained(local)
model = AutoModelForMultimodalLM.from_pretrained(local, dtype="auto", device_map="auto")

sys.path.insert(0, local)
from deep_projector import install_deep_projector
ckpt = torch.load(f"{local}/projector_final.pt", map_location=model.device)
deep = install_deep_projector(model, hidden=ckpt["config"]["hidden"],
                              n_hidden_layers=ckpt["config"]["mlp_layers"])
deep.load_state_dict(ckpt["state_dict"], strict=True)
# ... then apply_chat_template / generate / parse_response as above.

Notes

  • Audio format: 16 kHz mono. Resample first (e.g. librosa.load(path, sr=16000)).
  • Thinking: with enable_thinking=True the model emits <|channel>thought โ€ฆ <channel|> (including a <heard>โ€ฆ</heard> transcript of the audio) before the answer. processor.parse_response(...)["content"] returns just the final answer; strip any residual <heard>โ€ฆ</heard> if present.
  • Decoding: greedy (do_sample=False) is reproducible and used for all benchmark numbers below. For more varied generation use Gemma-4 sampling: do_sample=True, temperature=1.0, top_p=0.95, top_k=64.
  • Stop token: generation ends on <turn|> (the Gemma 4 turn terminator).
  • Long answers / runaway thinking: cap total length with max_new_tokens and optionally force-close the thought channel after a fixed budget by writing a small LogitsProcessor that forces the <channel|> token once N tokens have been generated (see deep_projector.py / the training repo for the reference ThinkingBudgetProcessor).

VoiceBench

Evaluated with the official VoiceBench protocol (greedy decoding, thinking enabled).

Subset Score
AlpacaEval 4.67
CommonEval 4.26
WildVoice 4.23
SD-QA (USA) 73.42
MMSU 85.20
OpenBookQA 93.63
BBH 87.10
IFEval 84.54
AdvBench 95.77
Overall 86.98

Intended use & limitations

  • Designed for English spoken questions/instructions โ†’ text answers.
  • Not a transcription service (though it transcribes internally); not intended for non-speech audio, speaker ID, or languages other than English.
Downloads last month
31
Safetensors
Model size
33B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support