Google Open Sources TorchTPU as Meta Runs TPUs in Production Inference

On October 20, 2026, Google announced at PyTorch Conference 2026 that TorchTPU, its native PyTorch stack for Google TPUs, is now open source on GitHub. The release makes TPUs a native option for PyTorch alongside GPUs and CPUs. The stack was co-developed with the PyTorch team at Meta. Google Cloud Blog reported the launch lets existing PyTorch workloads run on TPUs with minimal code changes, with developers debugging in eager mode and moving to production performance through torch.compile.

The change addresses a long standing friction point. PyTorch is the dominant framework for model development, but Google's TPUs previously required JAX or the older PyTorch/XLA bridge. For many teams, using TPUs meant rewriting code, learning a different runtime, or accepting a second class experience. TorchTPU is designed to remove that penalty by presenting the TPU as another PyTorch device.

The production claim gives the release more weight than a typical tooling update. Meta's PyTorch-TPU lead Zongwei Zhou said that SGLang-TorchTPU is running in production at Meta today, after a close three way collaboration between Meta, Google, and the SGLang team. Tom's Hardware reported on October 22, 2026 that Google's TPUs now serve AI in production at Meta through a new PyTorch native version of the SGLang inference engine.

The timing matters because AI hardware competition has been shaped by Nvidia's CUDA software ecosystem. Google's TPUs are among the few alternatives with years of production experience inside Google. The release tries to move competition from chip specifications to framework compatibility. It came one day after Amazon open sourced TorchNeuron, its native PyTorch backend for Trainium2 and Trainium3, according to The Frontier on October 21, 2026.

Key Facts

Google Cloud Blog reported on October 20, 2026 that TorchTPU is available as open source on GitHub and was co-developed with Meta's PyTorch team. The release makes TPUs a native PyTorch option alongside GPUs and CPUs, and Google positioned it around inference cost and latency as the primary bottlenecks for reasoning and agentic models. Developers debug in eager mode, then move to production performance through torch.compile.

Google Developers Blog reported on October 20, 2026 that Google launched native PyTorch backends for vLLM, developed with Inferact, and SGLang, developed with RadixArk, both built on TorchTPU. Both engines previously ran on TPU only through JAX based backends. The new versions preserve each framework's worker model, scheduler, KV cache manager, and API server while adding TPU implementations and custom Pallas kernels. Google said both engines are already running at scale in production with preview customers. Zhou called SGLang-TorchTPU a remarkably mature, PyTorch native serving stack that integrates into Meta's existing AI infrastructure for frontier workloads on TPUs.

Tom's Hardware reported on October 22, 2026 that Jana van Greunen, director of PyTorch engineering at Meta, said Meta cannot be constrained by a single hardware architecture and that TorchTPU enables Meta to deploy its PyTorch workloads to Google TPUs. Meta had used TPUs through the older PyTorch/XLA bridge from 2020 to 2023 before cancelling those contracts, according to SemiAnalysis. It remains unclear where Meta's production TPUs sit. Talks covered cloud rentals in 2026 and purchases in 2027, and Meta reportedly signed a multi-year rental deal.

Performance claims are specific. Inferact ran 16 Ironwood chips, Google's seventh generation TPU, with speculative decoding and served Moonshot AI's 2.8 trillion parameter Kimi K3 model at more than 700 tokens per second for a single user. Tom's Hardware reported the comparison with 452 tokens per second on a 16 GPU Nvidia GB200 setup. Kernels for Kimi K3 and Qwen 3.8 27B deliver roughly 1.4 to 2 times GB200 throughput at small batches without speculative decoding. Google also cited up to 52% Model Flops Utilization on Qwen 3 8B and stability across up to 8,192 Ironwood TPUs. Google Developers Blog reported a public SemiAnalysis InferenceX run dated October 12, 2026 that put Ironwood at the lowest overall cost per million tokens for Qwen 3.5 397B-A17B at 2.01 cents.

Hugging Face reported on October 20, 2026 that Transformers, Diffusers, and Kernels now run natively on TPU and that 96.5% of the tests that run on TPU pass. It filed 34 issues against the TorchTPU repository with 19 already fixed. At batch size 64, decoding was 21.5 times faster on Qwen3-30B-A3B and 12.8 times faster on Gemma 4 26B-A4B compared with the default implementation on a TPU v6e 8 slice. The Frontier reported on October 21, 2026 that TorchTPU uses a PrivateUse1 C++ backend, has roughly 450 ATen operators, ships as torch_tpu 0.2.0 on PyPI, and is licensed under an Apache 2.0 style licence.

Analysis

The immediate significance is not that a new backend exists. It is that a major hyperscaler has publicly committed a production inference path to Google TPUs while keeping PyTorch as the interface. What this really means is that Google is no longer asking PyTorch teams to become JAX teams before they can use TPUs. The old PyTorch/XLA path was good enough for experimentation but not for the default workflow of most model developers. TorchTPU changes the default by letting teams keep existing serving engines, schedulers, KV cache managers, and APIs. Production inference is a harder test than a benchmark because it requires reliability, integration with existing infrastructure, and predictable cost. Zhou's statement is therefore the most consequential fact in the announcement.

The evidence still has edges. Neither Google nor Meta disclosed where Meta's production TPUs sit, how many chips are involved, or which workloads run on them. Tom's Hardware reported that talks covered cloud rentals in 2026 and purchases in 2027 and that Meta reportedly signed a multi-year rental deal. A rental arrangement is not the same as a wholesale switch to TPUs, and it does not prove that Meta will buy Google hardware at scale. The performance numbers are also narrow. The 700 tokens per second result used 16 Ironwood chips with speculative decoding on one 2.8 trillion parameter model, while the 452 tokens per second comparison used 16 GB200 GPUs. Those are useful signals, but they are not a full comparison across training, serving, networking, and utilization. The Frontier review added that peak performance still requires per accelerator shape and sharding work.

The bigger picture here is that the competitive boundary in AI hardware is moving from instruction sets to developer experience. Nvidia's advantage has never been only raw throughput. It is the habit of developers who know that PyTorch code will run on CUDA with minimal friction. Google is trying to neutralize that habit for TPUs, and Amazon is pursuing the same logic with TorchNeuron for Trainium2 and Trainium3. The two releases do not end Nvidia's lead, and neither company claimed that they would. They do reduce the cost of trying an alternative. That is the first step in any serious challenge to a software moat.

Why It Matters

For buyers, the change is about choice and pricing. Teams that wanted TPUs previously had to accept a separate software stack or a bridge that added friction. TorchTPU lowers that barrier by making the TPU just another device in PyTorch. Hugging Face reported that Transformers, Diffusers, and Kernels now run natively on TPU and that Optimum TPU is no longer required. Google also said the new engines support automatic prefix caching, KV cache offload via TPU Sync, speculative decoding with MTP, EAGLE-3, DFlash, DSpark and n gram drafters, long context parallelism, disaggregated prefill and decode, multimodal vision language models, and FP8, FP4, INT8 and INT4 checkpoints. Model families include Qwen, DeepSeek, GLM, Kimi, GPT-OSS and Gemma 4.

The cost data is the part buyers will watch. Google Developers Blog reported that Ironwood had the lowest overall cost per million tokens at 2.01 cents for Qwen 3.5 397B-A17B in a public SemiAnalysis InferenceX run dated October 12, 2026. If that result holds across workloads, it gives cloud buyers a concrete reason to ask Nvidia suppliers for better pricing or to split inference between GPU and TPU fleets. It also gives Google a counter to the argument that TPUs are only attractive inside Google. Meta's production use is evidence that an external customer can run a PyTorch native TPU path. The caveat is that cost per token depends on model, batch size, latency target, and utilization, so a single benchmark cannot be treated as a universal price list.

Next Up

RadixArk said SGLang-TorchTPU will add DeepSeek V4.1 Flash next and targets Day 0 TPU support for new open models by the first quarter of 2027. The three companies will share more at SGLang Summit 2026 on November 12 and 13 in San Francisco. The next proof points are more production workloads at Meta, independent replication of the Ironwood versus GB200 serving results, and clarity on whether Meta's TPU use is a rental experiment or a longer term purchase plan. Amazon's TorchNeuron launch one day earlier ensures that the PyTorch native device race will not be a Google only story, and The Frontier reported that both projects are Apache 2.0 licensed. For buyers, the practical question is no longer whether PyTorch code can run on alternative accelerators. It is how much rewriting, tuning, and operational work remains after the first run.