My News Pipeline’s Best AI Model Isn’t The One I Actually Run

🔍 Read the full analysis: My News Pipeline’s Best AI Model Isn’t The One I Actually Run on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the little things that make your day delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

GPT-6 Astra scored highest in Thorsten Meyer’s test of AI models writing news articles, but his production pipeline runs GPT-6 Luna, which costs about one-hundredth as much per article. In a report published October 13, Meyer said he compared 71 model configurations on real news sources and found that the most expensive option did not clear a practical quality bar worth its added cost.

Meyer said the test ran from September 8 to October 13, 2026, using the production prompt from his news pipeline. Models generated English and German versions from two source texts: a 6,340-character product filing and a 619-character political news item. He described the shorter source as more representative of much of his production work. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, assessed the articles on source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with German-language quality also scored.

GPT-6 Astra scored 4.00 out of 5 in each language in the standard test and 3.93 across 12 articles in a follow-up run. Meyer reported perfect source-fidelity scores in the standard test. But Astra cost about $0.12 per article and took around 50 seconds. GPT-6 Luna scored 3.67 in English and 3.68 in German in the standard test, and 3.67 in the follow-up, at roughly $0.0012 per article and 17 seconds. Meyer said every Luna article in the reported run produced output his pipeline could use after its normal repair step.

The report also found no dependable quality gain from raising Luna’s reasoning effort. The highest setting cost about 12 times more and took about 12 times longer than the no-effort production setting, while its scores remained within the test’s noise range. Meyer said the test led him to remove two backup models from the news route: GPT-4.1 nano, which received the lowest source-fidelity score and produced invented quotations, and locally run gpt-oss-120b, which produced no valid articles in four attempts.

Why the Cheaper Model Runs

The results highlight a practical difference between a model’s highest measured quality and the model that makes sense for repeated production work. Meyer’s reported cost estimates put Astra at about 100 times Luna’s per-article price. For a high-volume publishing workflow, that price gap can outweigh a modest quality advantage if the lower-cost model consistently meets editorial requirements.

The test also points to a less visible risk in automated publishing: backup models can affect accuracy when a primary service fails. Meyer’s account says the two removed fallbacks produced invented quotations or output the pipeline could not use. Those findings are specific to his prompts and test sources, but they underline why a failover chain needs evaluation as well as the main model.

For readers, the report is a case study in why public benchmark rankings may not predict performance on a particular task. Meyer said Claude Opus 5.5, which he uses for software development and which served as one of the judges, scored below Luna as a news writer. That is his test’s result, not a general ranking of the models across other tasks.

How Meyer Tested the Models

Meyer’s comparison was designed around one production task: turning a supplied news source into a finished article using the same prompt as his pipeline. The 71 configurations included API and subscription models, free-tier models accessed through OpenRouter, and open-weight models running on his own hardware. The test covered English and German editions aimed at global readers and readers in Germany, Austria and Switzerland.

For the initial scoring, both judges saw the source text as well as the generated article. Meyer said the strongest configurations were then tested on 10 additional articles, mostly based on short sources. He reported 95% bootstrap intervals for the follow-up results and excluded outputs that were not valid after the repair step used in production. He also cautioned that, with only two articles in the initial test, score differences under about 0.3 points could be noise, making the larger follow-up run more useful for comparing close results.

The report says Meyer moved GPT-6 Luna into the production route and removed GPT-4.1 nano and gpt-oss-120b on October 7. It also reports that Claude Sonnet 5.5 came close to Luna’s initial scores but cost about $0.036 per article at an API-equivalent price. These comparisons concern the specific prompts, sources and settings in Meyer’s evaluation.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in his October 13 report

Limits of This Model Test

The findings are from one publisher’s pipeline, prompt and set of source materials. The initial comparison used only two source texts, and the follow-up covered 12 articles per strongest configuration; the report does not establish that the same model ordering would hold for other publishers, topics, prompts, languages or source formats. The judges were also AI systems, rather than an independent panel of human editors.

Meyer reports bootstrap intervals and describes the Astra–Luna gap in the follow-up as distinct, but the supplied material does not provide the full underlying article-level scores, judge-by-judge results or a detailed account of how disagreements were resolved. The claim that the weaker fallbacks invented quotations is attributed to the report’s judges and test outputs; the source material does not include those full outputs for independent review.

It is also unclear how Luna will perform over a longer production period or on a wider range of breaking-news sources. Meyer gives a cost and median timing from the past week for production, but does not provide a longer-term audit of factual errors, corrections or publishing outcomes.

Monitoring Luna in Production

Meyer’s reported setup keeps GPT-6 Luna as the production model after the October 7 removal of the two tested fallbacks. The next meaningful check will be whether Luna continues to meet the pipeline’s quality and output-validity requirements on a broader stream of real news sources, particularly the short-source material Meyer says is common in production.

The report does not announce a date for another benchmark or a planned model change. Any future comparison would need to weigh article quality against cost, speed and the risk of unsupported details, using the same production conditions if the results are to be comparable. Until then, the published conclusion is limited to Meyer’s test: Astra led on measured writing quality, while Luna was his chosen balance of cost and performance.

Key Questions

Why does Meyer use GPT-6 Luna instead of GPT-6 Astra?

Astra scored higher in Meyer’s test, but he reported that it cost about $0.12 per article, compared with roughly $0.0012 for Luna. He chose Luna because its scores were consistently good in his follow-up test at a much lower reported cost.

How many model configurations were tested?

Meyer said he tested 71 configurations between September 8 and October 13, 2026, using real news sources and his production prompt.

Did higher reasoning effort improve Luna’s articles?

Not reliably in the reported test. Meyer said score differences across the six settings were within the noise range; the highest setting cost about 12 times as much and took about 12 times longer than the no-effort setting.

Why were two backup models removed?

Meyer reported that GPT-4.1 nano received a source-fidelity score of 1.00 and that judges flagged invented quotations. He said gpt-oss-120b produced zero valid articles out of four and also generated invented named quotations.

Do these results show which AI model is best for every news publisher?

No. The findings apply to Meyer’s prompt, sources, languages and workflow. The report supports the case for testing models on the intended task, but does not establish a universal ranking for news writing.

Source: ThorstenMeyerAI.com