A CPU Inside an LLM: Zero-Gradient Compilation of Exact Circuits into Pretrained Weights
Abstract
Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid (>250times dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension (d to d+K) with d/(d+K) norm compensation, Base-1000 canonical BPE emission, and differential (+254,-256) tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed {+,-,times} circuits in constant TC^0 depth alongside 3-literal Boolean (100%, N=1{,}728; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic (N=1{,}200, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, le 0.036%) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, le 0.057%) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered >2K-token Tango splits (N=4{,}392; 0/11,134 OOD triggers).
Community
Why teach an LLM arithmetic when you can compile arithmetic directly into its weights?
TL;DR: From learning weights to programming weights. We introduce closed-form, zero-gradient compilation primitives that embed deterministic computation directly into pretrained LLMs, demonstrating signed multi-digit arithmetic (+, -, ×) and automated compilation of Boolean, Majority, and Parity circuits across Qwen 3.5 4B, IBM Granite 4.2 3B, and architectural audits on Gemma 4 E4B.
- Zero gradient updates. Zero training examples. Zero external tools. Computation is encoded directly into model weights and executed through unmodified
model.generate(). - Exact logic across >250× RMSNorm swings. Two-layer Mellin-GLU quadrature transforms unknown per-token normalization scaling into translation along a log-spaced gate grid, with a calibration-free variant.
- Attention-free circuit routing on Qwen 3.5. Zero of eight full-attention layers modified. Cross-token operands are routed entirely through
GatedDeltaNetblocks. - Less than 0.06% additional active weights. Compatible with INT4/INT8 quantized backbones and a compact FP32 circuit sidecar.
Key results:
- 100% evaluated ALU-prefix accuracy with FP32 circuit execution.
- 100% evaluated Boolean, Majority, and Parity accuracy.
- 98–100% fluent inline continuation, versus 0% for 12.8K-example LoRA fine-tuning.
- Zero false circuit triggers across 11,134 out-of-domain inputs.
- Natural-language noninferiority across all 14 preregistered evaluation splits.
- INT4 deployment with just 4.6 GiB / 8.1 GiB VRAM for Granite / Qwen.
What if adding new capabilities to neural networks didn't always require training them?
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models (2026)
- Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers (2026)
- Sketchpad: certified exact arithmetic with a small vision-language model (2026)
- Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures (2026)
- Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2 (2026)
- Deterministic, Reproducible Inference in Exact Integer Arithmetic, and Verifiable Distributed Inference as Its Consequence (2026)
- Behavioral Capacity Certificates for Quantized Language Models (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.26450 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper