İvme-Conversate-v3.1-Base

Conversate-v3.1 Logo

İvme-Conversate-v3.1-Base is a 25M-parameter AI model based on İvme-Conversate-v3-Base. It was trained through continued pretraining to fix v3's problem of having good grammar but weak knowledge. Though even with the increased knowledge and capabilities in v3.1, benchmarks show a decrease in grammar performance.

Conversate-v3 really wasn't the update I was expecting. It showed me that yeah, architecture can help at this size, but data matters just as much when your model is this small. Conversate-v3.1 basically just solves that. Instead of another architecture change, I decided to go with the v2 playbook. Just feed it more data. To my surprise, even though I expected the model to be oversaturated by now, it worked.

Architecture

Parameter Value
Architecture Decoder-only transformer, dense (Same as v3, nothing magic.)
Parameters 24,956,480
Layers 16
Hidden dim 320
FFN dim SwiGLU, 864
Attention heads 8, full attention (no GQA), head_dim 40 (It's a bit of an odd number for GPUs, which makes v3 and v3.1 less efficient on GPUs)
Attention variant QK-Norm + XSA (Exclusive Self-Attention, arXiv:2603.09078)
Context length 1024 tokens
Vocab size 16,000 (custom digit-atomic BPE)
Positional encoding RoPE (θ=10,000)
Normalization RMSNorm (pre-norm)
Embeddings Tied input/output
Biases None

Its parameter count isn't exactly identical to v3. The FFN got padded from 853 to 864 so it makes better use of GEMMs. This didn't need retraining or anything. v3.1 just got a little chunkier for higher speeds. Still sub-25M.

Explanation for a Few Things

Most of these are pretty standard, so I'll only explain the special parts.

XSA (Exclusive Self-Attention) is a little attention mechanism discovered by Apple. It makes it so that each token looks at its neighboring tokens instead of focusing on itself. This allows the model to spend more effort on actual token-to-token relationships instead of spending compute on the token it already sees and doesn't need to process as much.

QK-Norm makes it so that the query and key values don't go crazy during long training runs and start drowning out other values and messing things up.

Digit-atomic BPE makes the tokenizer give every digit its own unique token (so instead of clumping 123 together, it separates them into 1 2 3, even if the data suggests clumping them). This makes math easier since the model can see and understand the relationship between individual digits. Still, like v3 showed, it's not a magic gateway. You still need good training to actually utilize it well. Let's just say Conversate models aren't really warm with math.

So What Does This Model Output?

It's kinda like a very intelligent autocomplete. It has no instruction training (the type of training where you feed an AI model user queries and AI model answers so it can chat), so it won't actually answer your questions. Though you can try formatting your text to mimic questions (like "The capital of France is". In that case, the model completes it as "The capital of France is Paris").

v3.1 is more stubborn than v3, in a good way. It tends to stick to its facts instead of constantly going into loops.

But this won't compete with, say, a 100M model. v3.1 is very good, but it can't do complex multi-step jobs or actually be ready for real-life usage.

Measurements

Benchmarks

All benchmarks were run using lm-eval-harness. To make sure the readings were correct, v3 was also benchmarked in the same training environment. So expect these v3 scores to differ slightly from what might be on the model card or leaderboards.

Benchmark v3 v3.1 Direction
ARC-Easy 39.02 40.15 up
ARC-Challenge 24.32 25.00 up
HellaSwag 28.90 29.52 up
PIQA 57.67 59.25 up
ArithMark-3 35.20 37.20 up
BLiMP (67-task macro) 78.49 75.09 down

Supporting language-modeling numbers:

Benchmark v3 v3.1
WikiText-2 word perplexity ↓ 57.90 55.35
WikiText-2 byte perplexity ↓ 2.1362 2.1182

(You'll mostly only see byte perplexity in actual benchmarks, but just in case, I still included word perplexity.)

But BLiMP Is Down?

Yes, it fell by 3.4 points. Conversate-v3.1 genuinely has worse grammar than v3. The reason is pretty simple: I stopped prioritizing BLiMP. Older Conversate releases had pretty high BLiMP scores, but everything else came out disappointing. And trust me, in real usage, BLiMP shouldn't matter this much. Conversate-v3.1 is also still very close to its competitors, even with the lower BLiMP score.

Actually, there's a secret v3.2 I trained to fix this. But the BLiMP fix brought down the benchmarks I actually cared about (II of 11.72). So I decided to go with v3.1.

Training Thingies

Data

31.4B tokens in total, just over a single epoch.

Source Share Why
HuggingFaceTB/finemath (finemath-3plus) 32.5% mathematical reasoning to help math performance
HuggingFaceTB/smollm-corpus (cosmopedia-v2) 29.7% synthetic textbook explanations to boost knowledge
HuggingFaceFW/fineweb-edu (score ≥ 3.5) 14.0% general educational web text, added just in case. Weirdly enough, it helps
HuggingFaceFW/finewiki (en) 11.9% encyclopedic references for boring but valuable information
open-web-math/open-web-math 10.5% mathematical web text for even more math
SimpleStories/SimpleStories 1.4% narrative and causal chains so it's ready for conversation in any future instruct fine-tune

Generic web text wasn't really moving v3 anymore. So I decided to go with this mix, designed specifically to patch v3's weak points. As you can see, it lines up pretty cleanly with the benchmark scores. Less generic text likely caused the worse BLiMP score, but the higher-knowledge sources brought everything else up.

The Recipe

Setting Value
Optimizer Muon (body weights) + AdamW (embeddings, norms)
Muon lr 5e-4
AdamW lr 1e-5
LR schedule Warmup-Stable-Decay (WSD)
Warmup steps 500
Decay fraction 10% of training
Weight decay 0.1
Gradient clipping 1.0
Precision bfloat16 (fp32 master weights)
Attention PyTorch scaled_dot_product_attention
Batch 128 × 1024 tokens × 8 GPUs
Steps 29,900

Using Muon on the body weights and AdamW on the embeddings and norm gains follows the same recipe v3 was trained with. Muon tends to help with training costs and stuff, plus it's just safer to stick with it since v3 was trained with Muon too.

Decontamination

No benchmark text was trained on, and that was enforced mechanically rather than just promised.

The pipeline checked all the training data against the benchmark splits from every benchmark I used. It took a long time to finish, but it's a good way to show that v3.1 isn't benchmaxxed.

Hardware

Eight RTX 5090s. Training alone took about 2.6 hours. It was running at around 3.4M tok/s, if I remember correctly.

Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "IvmeLabs/Ivme-Conversate-v3.1-Base", trust_remote_code=True, dtype=torch.float32,
)
tokenizer = AutoTokenizer.from_pretrained("IvmeLabs/Ivme-Conversate-v3.1-Base", trust_remote_code=True)
model.eval()

inputs = tokenizer("Once upon a time, there was a", return_tensors="pt")
out = model.generate(
    **inputs, max_new_tokens=200, do_sample=True,
    temperature=0.8, top_k=40, pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

It needs trust_remote_code=True because of the custom architecture. Also, a few warnings: it has no KV cache, so it has to reprocess the context for every new token during generation. I don't think it'll matter too much for a 25M model, but just be aware.

And after 1024 tokens, your prompts will be silently truncated, so don't get confused if it suddenly starts missing details. It can't even keep 256 tokens cleanly, but still :P

Limitations

  • It's base only, so it can't chat
  • Definitely not AGI
  • BLiMP regressed
  • English only (I know the org is called İvme, but I'm still not sure how I'm supposed to fit Turkish into an SLM)

What's Next

To be honest? Definitely Conversate-v4 after this. Conversate-v3 and v3.1 have too many architectural inefficiencies and problems that gave me way too many headaches. I'll start from scratch.

Citation

@misc{ivme-conversate-v3.1-base,
  author       = {IvmeLabs},
  title        = {İvme-Conversate-v3.1-Base},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/IvmeLabs/Ivme-Conversate-v3.1-Base}
}

By Eren Ekşi

Downloads last month
-
Safetensors
Model size
25M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train IvmeLabs/Ivme-Conversate-v3.1-Base

Paper for IvmeLabs/Ivme-Conversate-v3.1-Base