LLM Savings Surge: Cost-Cutting Innovations
The LLM Cost‑Cutter
Weekly briefing on advances that shrink the cost of large language models (Nov 11–18, 2025)
Pricing announcements from AI labs this week
- No public token‑price cuts from OpenAI, Anthropic, Google, Meta, Mistral, or Cohere in the last 7 days. Google’s Vertex AI Agent Engine runtime remains on a single regional rate (effective Nov 6); no new per‑token changes were posted this week. (cloud.google.com)
— — —
1) Anthropic’s three‑way deal with Microsoft and Nvidia signals more capacity and competition
- What happened (Nov 18): Anthropic committed $30B of spend on Microsoft Azure compute; Nvidia plans up to $10B investment in Anthropic and Microsoft up to $5B. Anthropic says Claude will be available across all three major clouds and plans to scale to roughly a gigawatt of Nvidia‑powered capacity. Plain English: a lot more AI compute is being financed and plugged in. More supply usually pressures prices or at least improves availability. (reuters.com)
- Why it matters for the LLM economy: Bigger, multi‑cloud supply reduces wait lists and enables better “shopping” across providers. That tends to push down effective $/token and improves “intelligence per watt” as newer, more efficient GPU generations (Grace Blackwell, Vera Rubin) come online. (businessinsider.com)
2) States set the rules of the power game: Kansas and Missouri finalize large‑load tariffs
- Kansas (Nov 13): Regulators approved special rates for ≥75 MW customers: long‑term contracts, lower energy rate but higher demand/grid‑cost share, and stiff exit fees. Goal: attract data centers while shielding other customers. (axios.com)
- Missouri (Nov 14–17): The Public Service Commission approved a similar large‑load tariff (≥75 MW). It’s designed so big users pay their “fair share” of grid upgrades; staff had proposed a more complex alternative that the commission rejected. (kcur.org)
- Why it matters: Predictable, “cost‑causer pays” power tariffs derisk new sites and speed interconnections. As more capacity plugs into utility‑priced power (instead of temporary generators), the cost of running models falls and planning gets easier—compounding hardware and software efficiency gains. (kcur.org)
3) Power options multiply: oil majors and battery giants court AI data centers
- Chevron (Nov 17): Announced a 2.5‑GW gas‑fired plant in West Texas dedicated to a major data center, expandable to 5 GW, targeting final investment decision in early 2026. Plain English: cheap local gas + on‑site power could undercut grid bottlenecks and lowers transmission losses. (mrt.com)
- Tesla (Nov 14): Pitched Megapack batteries to smooth AI centers’ rapid load swings; claims big lifetime value and fast deployment versus traditional grid upgrades. Translation: batteries can cut costly demand charges and keep GPUs busy when the grid is jittery. (tomshardware.com)
- Why it matters: Firm, fast power unlocks capacity sooner. On‑site generation and storage reduce downtime and penalties, improving dollars per token and tokens per watt by keeping clusters utilized. (mrt.com)
4) Live signals from the GPU market: cheaper “good‑enough” and more posted prices
- SaladCloud now lists RTX 5090 around $0.27/hr (batch); 4090s ~ $0.18/hr. That’s hobbyist‑to‑startup‑friendly pricing for small‑to‑mid models and batch inference. (salad.com)
- CoreWeave’s public price card shows RTX PRO 6000 Blackwell Server Edition nodes at $20.00/hr and posted B200/H100/H200 rates, improving transparency. For many 7B–70B‑class deployments, these “Pro” SKUs can be cheaper than top‑end racks. (coreweave.com)
- Runpod lists B200 from $5.49/hr per GPU, helpful for bursty jobs with per‑second billing. (runpod.io)
- Why it matters: More transparent menus—especially for mid‑tier, cost‑efficient GPUs—let teams match model size and SLA to the cheapest viable silicon, reducing $/request without hurting quality for “non‑frontier” workloads. These savings stack with runtime improvements. (coreweave.com)
5) Research that directly cuts serving cost or energy this week
- Decentralized Speculative Decoding (Nov 13): Adapts draft‑and‑verify decoding for distributed clusters, turning network wait into useful work; reports up to ~2.6× speedups on HumanEval/GSM8K without retraining. Plain English: faster decoding across many nodes = fewer GPU‑hours per job. (arxiv.org)
- FlashEVA efficient attention (posted Nov 1; circulating in practitioner circles this week): Up to 6.7× higher throughput and 5× lower peak memory vs. standard attention after light fine‑tuning. That means smaller, cheaper instances can serve the same traffic. (arxiv.org)
- KVTC (Nov 3; new follow‑up results discussed this week): A transform‑coding method that compresses KV cache by up to 20× (higher in some cases) while preserving long‑context accuracy—cutting GPU memory pressure and enabling higher batch sizes. (arxiv.org)
- Why it matters: These are “software flips” that reduce GPU memory and networking overhead, boosting tokens per watt and per dollar. They compound with cheaper GPUs and better power pricing. (arxiv.org)
6) Cooling and facility efficiency: AI for thermal control promises quick wins
- New surrogate models for data‑center thermal management (Nov 13): Researchers show 3D ML surrogates can predict temperatures in milliseconds (vs. hours for CFD), enabling real‑time cooling adjustments and reporting ~7% energy savings. Plain English: smarter fans and airflow mean less electricity for the same compute. (arxiv.org)
- Generative design for heat sinks (Nov 12): A diffusion‑based approach designs heat sinks with up to 10% lower pressure drop at the same temps—reducing pump and fan power. (arxiv.org)
- Why it matters: Lower PUE and smarter cooling directly reduce the power line in your LLM cost stack; these improvements pair nicely with runtime/quantization gains to raise “intelligence per watt.” (arxiv.org)
7) Macro: depreciation debate hints at longer useful GPU life
- Investors argued this week about whether 6‑year depreciation overstates earnings. A prominent analyst pushed back, noting strong demand even for older A100/H100s. If GPUs last longer productively, your annualized capex per token goes down. Plain English: if “old” GPUs stay useful for years, the cost curve bends further down. (barrons.com)
— — —
What this means for your unit economics this week
- Shop mid‑tier and consumer‑class GPUs for non‑frontier jobs. Treat ~$0.18–$0.27/hr RTX 4090/5090 as a floor for evals, batch inference, and small fine‑tunes; use posted $20/hr RTX Pro 6000 Blackwell and ~$5.5/hr B200 when you need more memory or throughput. This mix can cut spend without visible quality loss. (salad.com)
- Re‑price sites in Kansas and Missouri. If you’re planning ≥75 MW campuses, these tariffs reduce bill uncertainty but require commitment; model their demand charges and minimum‑bill features in your TCO. Earlier interconnection and predictable rates usually beat temporary generation on $/kWh. (axios.com)
- Exploit software wins first. Pilot decentralized speculative decoding and KV‑cache compression on your heaviest inference paths; target jobs with multi‑node latency and long contexts. Expect fewer GPU‑hours per request. (arxiv.org)
- Ask vendors about cooling controls, not just chips. Fast thermal surrogates and better heat‑sink designs reduce facility energy. A few percent PUE gain at scale equals real dollars and higher tokens per joule. (arxiv.org)
- Watch capacity deals (Anthropic–Microsoft–Nvidia) for downstream price pressure. As more efficient Blackwell‑class fleets light up across clouds, expect improved availability and potentially better spot/batch pricing—especially for thinking‑time workloads. (reuters.com)
If you want, next edition can add a quick tracker of posted Blackwell/Hopper rental rates, new state large‑load tariffs, and fresh serving papers that move tokens/sec or watts/token.