🔍 Read the full analysis: What To Measure In AI: Lessons From Amazon’s Bets on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Amazon published engineering guidance on Oct. 27 warning that token counts and other AI-use metrics can distort incentives when used as targets. The memo says those metrics can still help teams spot patterns, but the company’s Nova and Anthropic bets show why usage should be judged against business and customer outcomes.
Amazon published new guidance for its software engineers on Oct. 27, warning that token counts and other measures of AI use can create waste when treated as performance targets. The memo says teams should judge AI by its effect on customers, the business and colleagues—a question sharpened by mixed results from Amazon’s own Nova models and its investment in Anthropic.
The guidance was written by Amazon’s most senior engineers as a revised role guideline for software engineers, and was published by the company. It says measures such as token usage, AI-tool adoption, code changes and reviews may be useful indicators, but are “deeply flawed” when disconnected from outcomes and used as targets. Token counts represent chunks of text processed or generated by an AI model.
The memo does not say teams should stop using AI. It says input metrics currently “correlate well” with important outputs, while warning that making them goals can increase costs and complexity. Its practical advice is to examine both low and high adoption: understand why some teams use AI less, and check whether heavy users are delivering better results.
Amazon’s own AI bets provide a wider test of that principle. Business Insider reported in July that most flagship Nova models had been moved to maintenance. Amazon’s lifecycle page lists Nova Canvas, Nova Reel and the first Nova Sonic as retired. By contrast, Amazon’s June 30 filing valued its Anthropic holdings at about $190 billion, against roughly $18 billion invested, according to the source report. That is a paper valuation, not a realized return.
Six months at Amazon
From token leaderboards to an outcomes memo
Amazon spent 2026 learning the same lesson twice: once with how its engineers use AI, once with its own AI models.
- May 2026Token leaderboards backfireFinancial Times (via The Decoder): staff give an internal agent tool pointless tasks to inflate token counts. Reported, anonymous sources.
- Late May 2026KiroRank “deprecated”Amazon spokesperson to Business Insider. SVP Dave Treadwell: “Please don’t use AI just for the sake of using AI.”
- 24 Jul 2026San Francisco AGI Lab site closesConfirmed by Amazon to GeekWire. Frontier model research continues under Pieter Abbeel.
- 28 Jul 2026Most flagship Nova models wound downBusiness Insider, people familiar: Premier, Omni, Reel, Canvas to “keep the lights on”. Amazon: models “remain one of the most important things we’re working on”.
- 14 / 30 Sep 2026Nova Sonic v1, Nova Reel, Nova Canvas reach end of lifeAWS Bedrock model lifecycle page. Requests to them now fail.
- 26 Oct 2026Anthropic files its S-1Amazon notes: $6.2 billion principal, 12% ownership cap that Amazon can waive. Anthropic commits at least $110 billion to Amazon through April 2036.
- 27 Oct 2026The memo: outcomes, not tokenmaxxingUsage and throughput metrics are “deeply flawed” as targets. “Tokenmaxxing is at odds with frugality.”
- 29 Oct 2026Amazon’s third-quarter resultsAnnounced by Amazon. Not yet published when this article went live.
Sources: Amazon engineering memo (27 Oct 2026); Business Insider (29 May and 28 Jul 2026); GeekWire (24 Jul 2026); The Decoder (12 May 2026, summarising the Financial Times); AWS Bedrock model lifecycle (read 28 Oct 2026); Anthropic S-1 (26 Oct 2026).
Powered by Thorsten Meyer AI
A Shift From Usage to Results
The memo matters because companies adopting generative AI are under pressure to show whether the tools improve work, not just whether employees use them. A target such as weekly adoption or tokens processed is easy to count, but it does not establish that software is better, customers are better served or costs have fallen. Amazon’s guidance explicitly warns that turning these measures into goals can produce higher costs and more complex systems.
Its recommended approach is to treat usage data as a diagnostic. Leaders should look at the 10th percentile of adoption to learn where tools are not helping, and the 90th percentile to test whether heavy use is paying off. The memo also asks leaders to use AI in their own production work and explain where it helped or did not. That puts responsibility on managers to assess the work rather than simply reward activity.
For workers, the guidance offers a qualified defense of professional judgment: if current AI tools do not help with a task, documenting why can be useful evidence rather than a failure. For businesses, the central test is whether AI changes outcomes enough to justify its expense and operational burden.
Products Worth Considering
Fitbit Versa Smart Watch – Black Aluminum, One Size (S & L Bands Included)
Oura Ring 4 Smart Ring – Silver – Size 8
BowFlex Results Series SelectTech 552 Adjustable Dumbbell
As an affiliate, we earn on qualifying purchases.
From Token Leaderboards to Guidance
The memo followed reports of employees using an internal AI agent for unnecessary tasks to increase token counts and rank higher on leaderboards. The Financial Times reported in May that Amazon employees were inflating usage of an internal tool called MeshClaw. Those accounts were based on anonymous sources and remain reported claims, not a company-confirmed account of every instance.
Amazon did confirm that a leaderboard called KiroRank had been “deprecated,” Business Insider reported at the end of May. The company described it as an informal tracker created by employees and said it was never intended to encourage AI use for its own sake. Senior vice president Dave Treadwell also told staff: “Please don’t use AI just for the sake of using AI.” The new guidance makes the broader principle part of an official engineering role framework.
Amazon’s model portfolio adds relevant context. The source report says most flagship Nova models were moved to “keep the lights on” status, citing Business Insider, while Amazon’s product lifecycle information records several early Nova products as retired. The Anthropic stake is a different kind of bet: Amazon invested in an outside model maker and does not control it. The reported valuation reflects a filing figure, not cash Amazon has received.
““Tokenmaxxing is at odds with frugality.””
— Amazon engineering memo
The Returns Are Still Unsettled
The memo does not provide a company-wide estimate of how much AI has improved engineering output, reduced costs or changed customer outcomes. It says usage measures correlate with outputs at present, but gives no figures in the supplied material to quantify that relationship or show whether it holds across teams.
The reported value of Amazon’s Anthropic holdings is a paper valuation; the material does not establish a realized gain or explain how that figure may change. The status of Nova products also varies by model, and the source does not provide a full inventory of which models remain actively developed. Amazon’s third-quarter results have not yet been reported in the source material, so their financial disclosures cannot be used here to measure the returns on its AI spending.
Amazon’s Results Are Due Oct. 29
Amazon is scheduled to report its third-quarter results on Oct. 29, one day after publishing the guidance. Those results may provide additional financial context, but the source material does not specify whether the company will break out AI revenue, model-development costs or productivity effects. Any such conclusions should wait for disclosures rather than be inferred from the investment figures alone.
For now, the memo sets out how Amazon wants engineering leaders to evaluate AI: track adoption to identify patterns, investigate both low and high use, and judge performance by results. Whether that approach produces measurable gains across the company remains to be seen.
Key Questions
What did Amazon’s new AI guidance change?
Amazon published a revised engineering role guideline saying AI usage measures such as token counts should not be treated as performance targets. It directs teams to focus on customer, business and team impact.
Is Amazon telling employees to stop using AI?
No. The memo says adoption measures currently correlate with important outputs, but warns that using them as targets can create costs and complexity. It asks leaders to examine whether AI use is producing better work.
What does “tokenmaxxing” mean in this report?
It refers to increasing the amount of text processed by an AI tool to raise usage counts or leaderboard rankings, rather than because the work requires it. Reports about Amazon employees doing this were attributed to anonymous accounts; Amazon separately confirmed that its KiroRank tracker was deprecated.
What is the difference between Amazon’s Nova and Anthropic bets?
Nova is Amazon’s own model family; the source says most flagship models were moved to maintenance and lists several early products as retired. Anthropic is an outside model maker in which Amazon invested. The reported value of Amazon’s stake is a paper valuation, not a confirmed cash return.
When will Amazon report its next results?
Amazon’s third-quarter results are scheduled for Oct. 29, 2026. The supplied material does not say whether the report will give separate figures for AI productivity, model costs or revenue.
Source: ThorstenMeyerAI.com
