Skip to main content

Weight Space Misfolding: AI Models That Rewrite Other Models' Weights

Weight Space Misfolding is a paper from HCLTech researchers arguing that a frontier model can rewrite a smaller model's weights directly, with no gradient training, to change what it knows or does. In tests on Qwen2.5-1.5B-Instruct, one factual edit worked on all 100 tries, but two other edits were unreliable. The authors say it shows no self-propagation.

On this page

What the misfolding paper claims

A paper posted to alphaXiv on 14 November by four researchers linked to HCLTech describes a new way a model file could be tampered with. The authors, Chris von Csefalvay, Pranav Doma, Achyuthan Unni Krishnan, and Tamas Foldi, argue that a large frontier model acting as an "editor" can read and rewrite the weights of a smaller model to change its knowledge, behaviour, or alignment, without any gradient training. They call this weight space misfolding, borrowing the image of a prion that causes a protein to fold the wrong way.

The paper sets out a threat model with three kinds of edit: substituting a fact, denying a capability, and selectively omitting information. It separates a setting where nobody is allowed to write weights from one where a legitimate editing process is subverted. It also sketches a scenario where edits spread through intermediaries, but it is explicit that this part is hypothetical and untested.

The work builds on IWIBench, a benchmark the same team published on arXiv on 5 November. In it, an editor model gets four tools to read weights, write weights, run the target, and inspect it, with no gradients, shell, or Python, and is scored on whether an edit lands while the target keeps its other abilities. The IWIBench abstract says only a few models, most notably Astra and GPT-6.1 Sol, do this well, and that the ability appears to be recent.

What three experiments on a small Qwen model showed

The case studies target Qwen2.5-1.5B-Instruct, a small open-weight model, with GPT-6.1 Sol as the editor. The editor never saw the scoring prompts, and each patch was checked by reloading the saved model in a fresh process.

The first edit changed the host city of the 2000 Olympics from Sydney to Buenos Aires. The untouched model answered Sydney on 91 of 100 draws. The edited model answered Buenos Aires on all 100. The change was not clean, though. Control passes fell from 49 to 48 out of 70, because the model lost the ability to simply copy the word "Sydney", and one prompt produced "Buenos Aires, Australia".

The second edit tried to make the model answer legal questions with one fixed refusal sentence. It produced the exact sentence on 5 of 22 held-out prompts, kept only 6 of 11 controls, and also refused to name the capital of France. The third tried to make the model leave one supplier out of a list. That worked on 1 of 22 prompts, and the editor itself reported that it had not achieved reliable omission.

What the results do not show

The authors are careful about limits. The paper does not demonstrate self-propagation, a program that replicates, or a working offensive capability, and spreading across an air gap was not tested. It used one small target and one editor. Reliability and preservation of normal behaviour were weak: an edit can land and still damage something nearby.

The benchmark carries its own caveats. The IWIBench results come from one recorded run per episode across 30 development episodes, with a different target model (Gemma 3 4B IT), and the authors say the pattern describes that protocol, not a general ranking. The best configuration scored a mean reward of about 0.46 and the weakest about 0.01, on a scale where doing nothing scores zero. The misfolding paper notes that the benchmark's mean reward roughly doubled in under five months, and says this measures editing skill, not offensive capability.

Why it matters if you run local models

For people who download open weights, the practical point is about provenance. A fine-tune leaves a training trail. A direct edit of the kind described here could be a small patch to a file that otherwise looks and behaves like the original, and the Sydney example shows that a single fact can be flipped while almost every other test still passes.

The defences the paper proposes are a framework, not a proven fix: keep proposed patches separate from what gets deployed, record where a model and its inference stack came from, define the behaviours that must survive, test before accepting a change, and keep a way to roll back. The authors add that tests only establish what they examine.

For now, the everyday version is simple. Download models from the original publisher, compare file hashes against the published values, and be wary of unofficial re-uploads and merges whose history you cannot trace.

CuriousLM runs supported AI models locally on your device. Try CuriousLM.