Nvidia's Biggest Customers Are Building Their Own AI Accelerators — and the New Metric Is Cost per Token

On 10 August 2026, Reuters reported — citing The Information — that Microsoft could unveil its next-generation Maia 300 accelerator as soon as September, and has been discussing substantial 2027 manufacturing capacity with TSMC. Microsoft confirmed that custom silicon remains part of its long-term AI infrastructure strategy but disputed that the reported production figures reflected the scale of the programme.

Label that accordingly: Maia 300 specifications, September timing and volumes are reported, not announced. What is already official is more than enough for the argument.

Inference, not training, is driving this

Training is a capital event. Inference is a utility bill that arrives every month, forever, and grows with product success. Once a model is in production, latency per token, throughput per watt and cost per token become the numbers that determine gross margin. Microsoft positions Maia 200 explicitly as an inference accelerator, which tells you where the pressure is.

The memory wall matters as much as FLOPS

Autoregressive decoding is memory-bandwidth bound far more often than it is compute bound. You are streaming weights and a growing KV cache through the chip for every token. This is why the spec sheets have shifted their emphasis.

• Maia 200: 216 GB HBM3e at 7 TB/s, 272 MB SRAM, TSMC 3nm, native FP8/FP4 tensor cores.
• Google TPU 8i: 384 MB on-chip SRAM, 288 GB HBM, 19.2 Tbps inter-chip connectivity, aimed at inference and reinforcement learning.
• AWS Trainium3: UltraServers connecting up to 144 chips via NeuronSwitch-v1 / NeuronLink-v4 all-to-all fabric.

FP4, FP8 and why AI stopped caring about FP32

Low-precision arithmetic is not a compromise so much as a discovery: transformer inference tolerates aggressive quantisation far better than classical numerical workloads. Halving the bits halves the memory footprint and the bandwidth needed to feed the array, and roughly doubles the achievable throughput per unit of silicon. Native FP8 and FP4 support in hardware — rather than emulation — is where a meaningful part of the efficiency gap now lives.

A chip is useless at scale without a fabric

Google says a TPU 8t superpod contains 9,600 chips. AWS connects up to 144 Trainium3 chips per UltraServer. At those counts, the topology, bisection bandwidth and collective performance determine whether you get anything like peak utilisation — which loops directly back to the optical interconnect story. Custom silicon and custom networking are the same programme wearing two badges.

The real moat may be the compiler

Silicon efficiency on paper is worthless if teams cannot get their model onto it in an afternoon. Google leans on Pathways and JAX plus native PyTorch support; AWS has Neuron; Microsoft has its own stack. Developer portability is the counterweight to vertical integration, because no customer wants their architecture permanently welded to one accelerator family. The vendor who makes migration boring wins more workloads than the vendor with the better die.

Custom silicon does not mean Nvidia disappears

Google announced infrastructure around NVIDIA Vera Rubin in the same breath as its eighth-generation TPUs. The emerging picture is heterogeneous: merchant accelerators for flexibility and frontier training, custom parts for high-volume, well-understood, margin-sensitive inference. That is a portfolio strategy, not a replacement battle.

A timeline worth keeping straight

• Dec 2025 — AWS Trainium3 announced.
• 26 Jan 2026 — Microsoft Maia 200 officially launched.
• 22 Apr 2026 — Google announces TPU 8t and TPU 8i.
• 10 Aug 2026 — Reuters reports a possible September Maia 300 unveiling and 2027 capacity discussions.
• Sep 2026 — possible Maia 300 unveiling. Unconfirmed as of 11 August 2026.

The question is no longer whether anyone can challenge Nvidia. It is whether the companies buying the most compute can integrate their way to a cheaper token.

Sources: Microsoft, "Maia 200: The AI accelerator built for inference" (26 January 2026); Google Cloud TPU 8t/8i announcement and technical deep dive; AWS EC2 Trn3 UltraServers; Reuters report on Maia 300 (10 August 2026).