🔍 Read the full analysis: Claude And GPT-6 Astra: Testing Safe AI Weight Editing At Home on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment used a code-based weight edit to reduce looping in the local model Ornith 1.5 35B-A3B. On the 10 published Lexassay tasks, the edited model completed all 10, compared with 4 on the same experimental setup before editing, but its legal score remained zero. The report describes one run per setting and says a separate GPT-6 Astra review found faults in the analysis.
A home experiment reported by ThorstenMeyerAI.com found that a weight edit reduced looping in the local model Ornith 1.5 35B-A3B, helping it complete all 10 published Lexassay tasks on the study’s comparison setup. Its legal score nevertheless remained zero, underscoring that finishing answers is not the same as producing accurate legal work.
The experiment was conducted from Wednesday evening to Thursday morning and was updated Nov. 9 to clarify that its figures refer to the 10 published tasks; the runs and answers are public on Lexassay. The author says a Claude Opus 5.5 session wrote code to run the model, identify a direction associated with looping, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. No training or new legal data was used.
The author distinguishes this procedure from the MIT researchers’ reported “weight-space programming” approach. In the home test, Claude did not read and edit the weights by reasoning over them directly; the intervention was directional ablation, a technique based on recording the model’s internal signals during runs and projecting out a selected direction. The report says OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, numbers and claims and identified faults. That review was not a second experiment.
On the same experimental setup, the edited model answered 10 of 10 tasks, up from 4 of 10. It met 7 of 50 checked criteria, compared with 3 before the edit, but its legal score remained zero. The report says the edited answers contained 55 wrong or invented citations. It also notes that using the model card’s recommended sampling settings without editing allowed the unchanged model to answer 9 of 10 tasks on its original setup, making the effect of the edit difficult to isolate from setup differences.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Complete Answers, Same Legal Score
The result illustrates a meaningful distinction for people experimenting with local AI: an intervention may improve how reliably a model produces an answer without improving whether that answer is true. That matters in legal research, where fabricated citations or incorrect statements can mislead users even when a response is fluent and complete.
The report does not establish that weight editing improves legal performance generally. Its evidence comes from one model, one small published task set and one run per setting. The unchanged model’s 9-of-10 result under recommended sampling also suggests that configuration choices may affect completion rates. The strongest supported takeaway is narrower: this edit appeared to address a looping problem in this test, while the benchmark found no improvement in the model’s legal score.
For readers considering similar techniques, the public answers and citation checks offer a way to inspect the results rather than relying only on a headline metric. The author explicitly says the experiment is not legal advice.
Products Worth Considering
Philips Sonicare 6500 Series Rechargeable Electric Toothbrush with 2 Brush Heads, White
Blueair Blue Pure 211i Max Air Purifier for Large Rooms
Bowflex SelectTech 840 Adjustable Kettlebell
As an affiliate, we earn on qualifying purchases.
MIT Work Prompted a Local Test
The experiment followed a report by four MIT CSAIL researchers—Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab—on what they call weight-space programming. The source article describes their work as using GPT-6 Astra to inspect and alter another model’s weights directly. In a Qwen example, the researchers traced looping to an internal direction and removed it; the source says the model’s maths score rose from 7.9% to 48.1%.
The MIT researchers’ finding, as quoted in the source, was that “Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.” The home experiment tested a related idea by a different method: if looping prevents a model from completing tasks, suppressing the loop might improve completion, but it should not supply legal knowledge that the model does not have.
Ornith was chosen because the report says its benchmark runs showed looping, unlike two other local models that appeared to lack the needed knowledge. The report describes Ornith 1.5 35B-A3B as a mixture-of-experts model with about 35 billion total parameters and roughly 3 billion active per token, based on its model card. Lexassay asks models 15 questions about German, U.S. and EU law; 10 tasks are published and five remain hidden. The reported results here cover only the published tasks.
“Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.”
— The MIT researchers, as quoted by ThorstenMeyerAI.com
Setup Differences Limit the Finding
The report presents one run per setting, a small development set and a loop detector that changed during the experiment. It also says Claude carried out the edit and Claude agents handled blind grading; GPT-6 Astra provided a separate review and found faults. The source does not specify in the supplied material what every fault was or how each was resolved.
Results from two local setups are not directly comparable. The report says the unchanged model answered 8 of 10 tasks in its published run but only 4 of 10 on the other setup used for the comparison; with recommended sampling settings, it answered 9 of 10 on its original setup. These differences leave the edit’s independent effect on completion uncertain. The report also says three graders came from the same model family, which limits the independence of scoring.
The reported zero legal score and citation problems indicate that the intervention did not establish reliable legal performance. The source does not provide evidence that the method will work on other models, other benchmarks or the five hidden Lexassay tasks. The findings are a small-scale experiment, not a general demonstration of safe or dependable weight editing.
Public Runs Allow Further Checking
The immediate next step is scrutiny of the public Lexassay runs, answers, grades and citation checks, which the report says are available online along with a method page. Reviewing those records can help readers assess the completion counts, the legal grading and the errors identified in the separate code review.
The source does not announce a follow-up experiment or say when another evaluation will take place. Further tests would be needed to determine whether the completion change persists across repeated runs, matched sampling settings, additional models and tasks not used in development. Until such evidence is available, the result remains a report of one limited test: the model finished more answers on one setup, but its reported legal score did not improve.
Key Questions
What did the experiment change?
Code written during a Claude Opus 5.5 session removed a loop-associated direction at half strength from the output weights of 27 of Ornith’s 40 layers. The report says the model was run and measured; an AI did not directly read and edit the weights as text.
Did the edit make the model better at law?
The report says no legal-score improvement was recorded: the score stayed at zero. The edited answers met 7 of 50 checked criteria and included 55 wrong or invented citations, according to the article.
What does 10 of 10 represent?
It is the number of the 10 published Lexassay tasks the edited model answered on the comparison setup. The report says the unedited model answered 4 of 10 on that setup. The benchmark has five additional hidden tasks, which are not included in these figures.
Was this the same method used in the MIT research?
No. The source describes the MIT work as direct weight editing through reasoning about weights. The home experiment used directional ablation: it ran the model, measured an internal direction associated with loops and removed that direction from selected weights.
Can the result be treated as proof that weight editing is safe?
No. The report describes one experiment with one run per setting, setup differences and a small task set. It does not establish safety or reliable performance across models or uses, and it says the experiment is not legal advice.
Source: ThorstenMeyerAI.com
