Papers
arxiv:2611.11937

Attention at Absolute Zero

Published on Nov 17
· Submitted by
Neo Mohsenvand
on Nov 18

Abstract

Attention scores candidate supports of an answer, which we call witnesses, and reads the scores at a temperature. We build a higher-order attention layer whose state lives on pairs of items: for a pair (a, b), each third item k is a witness judged through its legs (a, k) and (k, b). It reads witnesses at zero temperature, keeping the best and the number tied with it, passes on only bits comparing such counts, rebuilds its pair state from the input and these bits at every loop, and is trained from answers alone. Each choice removes one way soft attention fails at size: dispersed weight, unresolved gaps between learned scores, and a drifting soft state. In maximum retrieval it is right on at least 99.95% of sets 1,000 times larger than in training, on all ten seeds; trained with a soft count, the same network falls to 42% at 4,096 items. On CLRS-30 without hints, a general form of the layer (half its channels read at zero temperature, numbers embedded, state carried) beats the best published model, which uses hints on some tasks; the full cold layer is exact on every test input of 9 of the 12 tasks whose answers depend on numbers only through comparisons. From yes-or-no answers it decides 2-SAT and two-colouring exactly at 500 variables and, as our theory predicts, not parity equations. Beside a frozen 7B language model it chains family relations exactly at 1,000 statements after training on 16, where the model alone falls to 44%; finite checks on the weights prove every block that learned correct at every size, on acyclic contexts and given what the model reads. On CLUTRR it is right on every story of five to ten relations given the graphs, learning the 62 rules exactly; from text, with the model frozen, no hand-written rule and only a small read-in trained, it lifts an open 7B model from 33% to 65%, above every system we found without hand-written rules except frontier reasoning models.

Community

Paper author Paper submitter

A higher-order attention layer whose state lives on pairs of items. Each pair (a, b) judges every third item k through its two legs (a, k) and (k, b), reads these witnesses at zero temperature (the best witness and the number tied with it), passes on only comparison bits, and rebuilds its pair state at every loop. It is trained from answers alone, without hints.

  • CLRS-30: one block shared by all 30 algorithms averages 97.20% at 64 items, above the best published model even with hints (96.64%).
  • Maximum retrieval: at least 99.95% on every one of ten seeds at 16,384 items, 1,000 times the training size.
  • 2-SAT and two-colouring: exact at 500 variables, learned from yes-or-no answers.
  • Beside a frozen 7B language model: chains family relations exactly at 1,000 statements after training on 16, and every trained block that learned is proven correct at every size (on acyclic contexts, given what the model reads).

Try all 30 algorithms in your browser: https://neovand.github.io/cold-attention
Code: https://github.com/NeoVand/cold-attention
Models: https://huggingface.co/mohsenvand

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.11937
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 10

Browse 10 models citing this paper

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.11937 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.11937 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.