An output-head experiment · 17 November 2026

Does weight-space programming need to touch the hidden layers?

I tried leaving Qwen’s internal layers alone and editing only three rows that score its next token. The model reached 55.2% on MATH-500. Then I looked at the runs that still would not stop.

55.2%276 / 500 MATH-500
3output rows changed
0transformer tensors changed

In Sparks of Weight-Space Programming, Astra improved Qwen3.5-0.8B by editing two internal feed-forward layers. Nathan’s follow-up showed how a linear program could find a related edit. My experiment asks a narrower question: how much of the result can we recover at the model’s final output layer?

A reasoning model already has a token for ending its scratchpad: </think>. I fitted small linear classifiers to the model’s final hidden state, then folded their weights into the rows for that token and two spellings of Wait. This changes when it stops or says “Wait” while preserving every original transformer tensor. The edited weights run without a separate classifier at inference time.

On the full 500-question benchmark, this candidate scored 55.2%, alongside 52.2% for the selected published Astra-edited reference run. The runs use different engines, so this is evidence worth investigating—not proof of a better method. The more surprising result was where the remaining loops went.

Compare all 500 questions ↓Replay successes and failures ↓

Three output rows. 500 math questions.

All rows below cover the same 500 MATH-500 questions. Each uses one selected answer per question. The references come from external runs; this is a benchmark comparison, not a controlled same-engine experiment.

MATH-500 · thinking enabled · 32,768-token limit
ModelCorrectAccuracyDirect weight change
Unedited Qwen3.5-0.8Ba35 / 5007.0%None
RikiyaT / Astra editb261 / 50052.2%Feed-forward layers 15 and 19
Output-head probe graft276 / 50055.2%Three output rows

a Sample 0 of the original author’s full unedited evaluation, not the 25-question smoke test. b Sample 0 of the independent Hugging Face rerun, the strongest of its four global sample indices. The same index is used for every problem. Different engines and random draws remain confounders.

The observed lead is three percentage points. There are 46 questions only our sample gets right and 31 only the reference gets right. An exploratory paired bootstrap gives a difference interval of −0.4 to +6.4 points; descriptive McNemar p = 0.110. This does not establish superiority or equivalence.

Our mean total output is 15,829 tokens. On the 20 questions with published reference sample text, ours averages 15,340 versus about 21,638 retokenized reference tokens: 29.1% fewer on that subset. That is not a 500-question token-saving estimate.

GSM8K and IFEval are follow-up evaluations. Their final results are not included in this release. Frozen results

Watch the output, including the failures.

These are saved generations, replayed by token position—not live inference. The examples include a shorter success, an extra correct answer, and two failures. The complete 500 outputs are downloadable.

Loading…
0 tokens
Unedited
RikiyaT edit
Output-head graft

Reference token positions are approximate: the published text was retokenized. Playback speed is a reading control, not measured inference throughput.

Read a signal. Edit its output row.

Before choosing a token, the model produces a vector called the hidden state. The output layer turns that vector into a score for each token. One row scores </think>; other rows score Wait and Wait.

Transformer blocksAttention, FFNs, norms299 tensors unchangedOriginal model weightsHidden state h1,024 numbersOutput scores</think> + stopping signalWait − looping signal Wait − looping signalTransformer blocks unchanged299 original tensors preservedHidden state h1,024 numbersOutput scores</think> + stopping signalWait − looping signal Wait − looping signal
Only these three output rows differ. The input embedding stays intact.

A linear probe is a classifier that scores this state with a dot product. If its weights are g, its score is gTh. Adding the probe to an output row builds that same calculation into the token’s existing score. No separate probe runs during generation.

ℓ′ⱼ = (wⱼ + αg)ᵀh = ℓⱼ + αgᵀh

For this 0.8B candidate, the stop row combines a re-checking probe with a loop-boundary probe. A third probe suppresses the two Wait tokens when the state resembles looping. It is an extension of post 1’s single-row stopping edit, not the exact same one-row model.

The actual three-row recipe
ℓ′end = ℓend + 12 gdoneᵀh + 24 gboundaryᵀh
ℓ′Wait = ℓWait − 4 gloopᵀh

The second equation applies to both spellings of Wait. Coefficients were selected using development problems, not benchmark answers. This is still a weight edit; the intervention is at the output head rather than inside the transformer blocks.

At a fixed hidden state, the logit changes are directly calculable. Top-k/top-p filtering and subsequent generation prevent an exact reconstruction of every edited trajectory from an original trace. All scores here come from live generation.

A concrete example: 25 choose 2.

One fitting problem asks for C(25, 2). The base trace calculates 25 × 24 / 2 = 300, states “Final Answer: 300,” and then keeps going to the 16,384-token fit limit.

Calculation: 25 × 12 = 300.
Final Answer: 300.
…followed by further checking.

A fitting example, train/2006, not a held-out benchmark question. Recorded excerpt.

  1. Record the base model. Save its final hidden states while it reads its own reasoning traces.
  2. Label useful distinctions. Mark paragraph boundaries after an answer appears, and repeated-line regions; use other reasoning states as negatives.
  3. Fit the directions. Shrinkage linear discriminant analysis finds directions separating those groups. It is a linear-algebra fit with labels, not gradient descent or base-model retraining.
  4. Calibrate, then freeze. Fit on 80 MATH-training questions, select strengths on 16 separate training questions, and keep the chosen weights fixed throughout the benchmark.

The output location is easy to specify: known token rows, no search over internal layers. The data work is not trivial. Collecting traces can take hours; the matrix solve is the quick part. An answer appearing in text is also an imperfect label—it does not prove the model has finished checking it.

Same problem. A different place to intervene.

RikiyaT’s Astra experiment removes a loop-associated direction from two internal feed-forward matrices. Nathan’s LP follow-up finds a related sparse internal edit. Here, the transformer computation is kept intact and the final token scores are edited directly.

Both families use activations and can be state-dependent. “His is constant; ours is conditional” would be too broad: Nathan measured a near-constant effect for a particular LP edit. An ordinary checkpoint with no runtime hook is also a feature of his method, not unique to this one.

The practical advantage I’m testing is a more mechanical recipe: record final states, fit a classifier, write a few known output rows. There is no search over which internal layers to alter. The similar observed scores make that recipe worth testing further; fewer edited weights alone do not guarantee fewer side effects.

Earlier 4B, DeepSeek and agent observations

The earlier single-row method was tested separately on Qwen3.5-4B and DeepSeek V4.1 Flash. On eight paired 4B math questions, both models got 8/8 while total tokens fell 54%. DeepSeek’s six shared solved questions used about 40% fewer total tokens. Agent examples showed selective thinking around planning, errors and new information.

Those are small exploratory studies with configuration caveats, not proof of universal transfer. Our local DeepSeek internal-layer interventions failed to shorten reasoning, but they were not a controlled reproduction of RikiyaT’s exact method. I cannot conclude his method fails on DeepSeek.

The algebra does not depend on a particular model size. Useful signals, calibration and quality do. Each model needs its own fit. Read the original experiments and limitations.

Related work already includes probe-guided stopping, activation directions controlling reasoning length, and NoWait token suppression. The contribution here is an output-weight implementation and its measurements, not the invention of early stopping.

Ending thinking is not enough.

The largest failure became clear only after the full run. The edited model closed thinking in 163 of the 165 capped cases, then continued in the answer. Across 407 responses it emitted 1,348,952 extra closing tags, around 17% of all generated tokens. The model has no hard phase gate; its stop boost can remain active after the first close.

The grader does not automatically reject capped runs. It extracts the last boxed answer after the last close, or from the full response if thinking never closed. An audit found no correct boxed answer anywhere in the 165 capped responses. Unboxed candidates may still go uncredited under this protocol.

A cheap 8-gram detector, applied to saved outputs, flagged all 165 capped runs and 206 of 335 completed runs. That is alert coverage, not independently annotated precision/recall. It does not establish that forcing a stop at the alert would produce the right answer.

Completed checks, open controls, and selection caveats
  • Done: complete MATH-500; separate fitting/development IDs; frozen weights; exact tensor comparison; sampler checks; capped-answer audit; a prefix-only repetition diagnostic; published raw results.
  • Still missing: tuned penalty/sampling controls, avg@8 or larger repeated draws, independently labeled detector precision/recall, and systematic batch-invariance checks.
  • Presence penalty 1.5 is enabled. That is one setting, not a calibrated penalty sweep. A fixed seed does not remove serving noise.
  • The development set was reused for model selection. A fresh 16-question check scored 8/16 for this candidate versus 9/16 for the previous graft. This negative result is retained.
  • The earlier p = 0.0078 referred to token reductions on eight exploratory Qwen-4B pairs, not an accuracy improvement or a test that excludes sampling and serving noise.
  • Qwen’s tied embedding/head requires a new output matrix: roughly 0.51 GB extra. There is no extra transformer pass, but it is not zero memory overhead.
  • The current three-row model can change non-stop token choices. It must not be described as the old “only one logit changes” model.

Analysis snapshot · Audits and provenance

Reproduce this run.

The GGUF is an experimental artifact, including the failures above. Start with unedited Qwen3.5-0.8B. All 299 original tensors were checked byte-for-byte; only the separate output head differs. The portable rebuild matches the evaluated file’s SHA256.

hf download vaibhavrpi/Qwen3.5-0.8B-probe-graft-GGUF \
  Qwen3.5-0.8B-BF16-probe-graft.gguf --local-dir .
llama-server -m Qwen3.5-0.8B-BF16-probe-graft.gguf \
  -ngl 99 -fa on -c 278528 -np 8 -b 256 -ub 256 --jinja
Sampling, seeds, environment and scoring
Samplingtemperature 0.6; top-p 0.95; top-k 20; min-p 0
Penaltiespresence 1.5; frequency 0; repetition 1.0
Seed & samplesSeed 0 for every request; one sample per problem
Budget32,768 new tokens; thinking enabled
Sampler orderpenalties → top-k → top-p → temperature
Serving8 slots × 34,816 context; batch and microbatch 256; flash attention; all layers on GPU
HardwareRTX 4060 Laptop, 8 GB; single GPU
SoftwareUbuntu 26.04; CUDA 13.2.78; driver 595.71.05; Python 3.14.4 generation, 3.12.12 fitting/scoring
Engine source3549c274dc22001d9118713fa56a10085b85de3b, plus included sparse presence-penalty optimization
Base revision2fc06364715b967f1860aea9cf38778875588b17
Scoringmath-verify 0.9.0; IFEval strict prompt-level checks

The llama.cpp penalty window is 33,792 tokens and includes the prompt; the reference vLLM penalty covers generated tokens. The sparse implementation preserves penalty arithmetic; numerical checks are not a claim of zero serving noise.

For math, the exact user suffix is \nPlease reason step by step, and put your final answer within \boxed{}. Dataset revisions, package lock, environment manifest and reconstruction commands are in the repository. All benchmark generation is live; playback is for inspection.