🔍 Read the full analysis: GPT-6 Astra Tweaked Qwen’s Weights—Here’s What Happened To JSON Tabs on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
An experiment run by OpenAI’s GPT-6 Astra as a Codex agent changed one matrix in Qwen3-4B-Instruct-2507. On a frozen test of copying special characters into JSON, exact answers rose from 49 of 120 to 70 of 120, with 21 cases fixed and none broken. The result is limited to synthetic cases; a separate table-extraction score remained zero, and no independent review has been completed.
An experiment conducted by OpenAI’s GPT-6 Astra as a Codex agent raised Qwen3-4B-Instruct-2507’s exact score on a test of copying special characters into JSON from 49 of 120 cases to 70 of 120. The reported gain came from a calculated edit to one model matrix, not gradient training, and applies to a narrow set of synthetic tests rather than JSON tasks generally.
The test asked the model to place short text containing characters such as tabs, line breaks, quotation marks or backslashes into a JSON object. An answer counted as correct only if it was valid JSON and decoding the relevant field reproduced the original text character for character. In JSON strings, a tab must be represented with the escape sequence \t; copying a raw tab or changing the text counted as an error.
According to the experiment report, the edited model answered 70 of 120 final cases exactly, compared with 49 for the original. That is a gain of 21 cases, or 17.5 percentage points. The report says there were 21 cases fixed and no cases broken in the comparison. The test set was frozen before the result was measured, and the figures were updated on November 9 to refer to the 10 published Lexassay tasks; the runs are publicly available on lexassay.com.
GPT-6 Astra selected and applied a calculated rank-one update to the output projection of the model’s final MLP layer. The report says the other 397 stored tensors remained bit-identical to the original. It also records that a more explicit prompt did not help in the initial screening: both the original prompt and an escaping-focused version scored 16 of 32 cases.
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
A Narrow Gain in JSON Fidelity
The result matters because an exact-character error can make structured data unreliable even when the output looks close to correct. A tab changed to spaces, omitted, or inserted without escaping can alter a field or cause a downstream parser to reject a record. The experiment tests a concrete failure mode that can affect data passed between software systems, including forms, logs and tool outputs.
But the score does not establish a broad improvement in JSON handling. Fifty of 120 cases remained wrong, and the report says a separate task—extracting a table row—still scored 0 of 30. The measured benefit is therefore confined to this particular copying task and its test set. Whether the edit helps on real documents, other prompts or other models has not been shown.
Products Worth Considering
Blueair Blue Pure 211i Max Air Purifier for Large Rooms
Bowflex SelectTech 840 Adjustable Kettlebell
Philips Sonicare ProtectiveClean 5300 Rechargeable Electric Toothbrush with 3 Brush Heads, Black
As an affiliate, we earn on qualifying purchases.
The Clue From an Earlier Run
The experiment was the third step in a week of work on editing model weights. In a prior attempt, GPT-6 Astra tested a different task on the same Qwen model: extracting five fields from synthetic business messages. An attention-head edit appeared promising during validation, but both the original and edited versions scored 97 of 200 on the frozen final test.
That run also surfaced the JSON escaping problem: the report says both versions produced 47 invalid answers, mostly by copying raw tab characters into JSON strings. The third experiment followed that clue and targeted the known failure rather than searching broadly for weaknesses. The series followed an earlier experiment in which Claude edited a different local model; GPT-6 Astra later reviewed that work and identified technical faults. Claude was not involved in the Qwen edit described here, although a Claude agent drafted the article from the records and checked the figures.
“The starting hint for experiment 3 came from the previous attempt.”
— The GPT-6 Astra experiment report
Limits of the Published Test
The reported test cases are synthetic, and the result has not yet been checked by an independent reviewer. The published record does not establish whether the improvement generalizes to natural documents, different JSON formats, other model versions or production data. It also does not explain why the specific matrix edit helped, beyond describing how it was calculated and applied.
The outcome is not a general certification of the model’s JSON output. The edited model still missed 50 final cases, and its table-row extraction result did not improve from 0 of 30. The report’s claims about unchanged tensors and the score comparison are attributable to the experiment records; independent replication would provide a separate check.
Independent Checks and Broader Tests
The runs and updated figures are public on lexassay.com, giving other researchers and developers a basis to inspect the test and compare the original and edited model. The next useful checks would be an independent review of the experiment, replication of the result, and testing on cases beyond the synthetic set. No further review or follow-up result is reported yet.
Until those checks are available, the supported conclusion is limited: one targeted weight edit improved exact copying on one frozen JSON escaping test, while leaving substantial errors and a separate extraction task unresolved. Whether the approach can produce reliable gains across other tasks remains open.
Key Questions
What changed in the Qwen model?
GPT-6 Astra applied a calculated rank-one update to the output projection of Qwen3-4B-Instruct-2507’s final MLP layer. The report says the other 397 stored tensors were unchanged.
How much did the JSON test score improve?
The edited model scored 70 of 120 exact answers, up from 49 of 120. The report describes 21 cases fixed and none broken, a gain of 17.5 percentage points on that test.
Does this mean Qwen now handles JSON reliably?
No. The result covers one synthetic copying task. Fifty of 120 final cases were still wrong, and table-row extraction remained at 0 of 30 in the reported separate task.
Has another team verified the result?
Not yet, according to the source report. It says the experiment has not had an independent review, although the runs and updated figures are public on lexassay.com.
Source: ThorstenMeyerAI.com
