LLM Savings Surge: Cutting Costs This Week!

The LLM Cost‑Cutter

Weekly briefing on advances that shrink the cost of large language models (Oct 28–Nov 4, 2025)

TL;DR: This week’s real cost movers are (a) fresh, public GPU-rental prices for Blackwell that keep drifting down, (b) a big Microsoft capacity deal that should ease scarcity, (c) U.S. regulators formally pushing to speed up data‑center grid hookups, and (d) software/runtime updates that squeeze more tokens per watt. One notable counter‑trend: new runtime charges from cloud AI platforms in certain regions.

1) Public GPU prices: Blackwell competition intensifies

  • Lambda’s live page lists 8×B200 at $4.99 per GPU‑hour on demand, with multi‑week cluster terms “as low as” $3.79/GPU‑hour; H100 shows $2.99/GPU‑hour. This page was updated this week. Plain English: market‑rate reference points for negotiating and planning just got clearer. (lambda.ai)
  • Runpod shows B200 available on secure cloud at $5.69/hr, plus new per‑second serverless rates: B200 at $0.00190–$0.00240/sec (active vs. flex). Plain English: more providers, more transparent pricing, finer‑grained billing. (runpod.io)
  • CoreWeave’s catalog (updated last week) highlights B200 and new Blackwell RTX PRO 6000 server editions, signaling broader Blackwell availability across instance types. Plain English: more SKUs means more ways to right‑size spend. (github.com)

Why this matters for costs and intelligence per watt

  • More posted prices + more supply generally push $/GPU‑hour down. Pair these with better runtimes (below) and you get fewer dollars and watts per useful token. (lambda.ai)

2) Big capacity: Microsoft signs $9.7B, five‑year deal for GB300 racks

  • Microsoft inked a $9.7B contract with IREN for Nvidia GB300 systems, deploying ~200 MW of liquid‑cooled capacity at IREN’s Childress (TX) campus through 2026. Plain English: more cutting‑edge inference/training gear will come online without Microsoft building all of it itself. (reuters.com)

Why this matters for costs and intelligence per watt

  • More near‑term capacity reduces scarcity premiums on GPU time; GB300 also claims sizable per‑watt performance gains versus GB200, improving tokens per watt as it ramps. (tomshardware.com)

3) Policy tailwind: DOE pushes FERC to fast‑track large‑load interconnections

  • On Oct 23 (covered over the past week), the U.S. Energy Secretary asked FERC to initiate a rulemaking to standardize and speed grid hookups for big loads like AI data centers, with a draft ANOPR and an April 30, 2026 response date proposed. Plain English: less waiting for power means data centers come online sooner and cheaper. (gibsondunn.com)
  • Related: PJM (the largest U.S. grid operator) published progress updates and continues queue reforms to shorten interconnection timelines. Plain English: the biggest queue is clearing faster, which lowers project risk and cost. (insidelines.pjm.com)

Why this matters for costs and intelligence per watt

  • Faster, more predictable power access lowers financing and temporary‑generator costs. Over time it shifts compute to cleaner, cheaper grid power—improving $/kWh and watts per token. (gibsondunn.com)

4) Runtime and kernel upgrades: more tokens on the same watts

  • NVIDIA’s TensorRT‑LLM 0.21.0 adds new quantization paths (w4a8 MXFP4/FP8), chunked/sliding‑window attention, prompt‑lookup speculative decoding, and KV‑cache transfer improvements—aimed squarely at higher throughput and lower latency on Hopper/Blackwell. Plain English: the same GPUs output more useful tokens per unit time and energy. (nvidia.github.io)
  • PyTorch 2.9 shipped recently with performance and hardware enablement improvements (including Arm), reducing overhead in compiled graphs and decoding paths. Plain English: the common framework many teams use just got a bit faster and broader. (pytorch.org)
  • vLLM’s public Q4 roadmap (updated six days ago) focuses on cutting CPU overhead and improving frontier‑cluster serving—areas that often bottleneck throughput/$ at scale. Plain English: serving stacks keep maturing for big clusters. (github.com)

Why this matters for costs and intelligence per watt

  • These optimizations increase tokens/sec without new hardware, multiplying savings when combined with cheaper GPU‑hours and better batching/scheduling. (nvidia.github.io)

5) Fresh research: faster “thinking” without extra watts

  • SpecDiff‑2 (Nov 1) proposes a diffusion‑based drafter aligned to an autoregressive verifier; reports up to +55% tokens/sec over strong baselines and up to 5.5× over standard decoding at no accuracy loss (lab results). Plain English: better speculative decoding = same answers, fewer milliseconds and joules. (arxiv.org)

Why this matters for costs and intelligence per watt

  • As these methods land in mainstream servers (TRT‑LLM, vLLM, etc.), you get cheaper, faster generation per request, compounding with hardware and price drops. (nvidia.github.io)

6) Platform pricing changes you should know

  • OpenAI: AgentKit/Agent Builder billing began Nov 1, 2025 (storage metering and other platform line items). Plain English: some “free” prototyping surfaces now meter usage—plan budgets accordingly. (openai.com)
  • Google Cloud: Vertex AI Agent Engine runtime will start charging in more regions on Nov 6 (Singapore, Melbourne, London, Frankfurt, Netherlands). Plain English: new line items can raise total cost of ownership for “agent” apps even if raw model token rates are unchanged. (docs.cloud.google.com)

Why this matters for costs and intelligence per watt

  • API bills are the sum of model tokens and platform runtime add‑ons. Keeping an eye on these fees—and shifting workloads to cheaper runtimes when possible—directly lowers $/request. (cloud.google.com)

7) Quick market signals (context for the next few weeks)

  • More Blackwell SKUs are surfacing (e.g., RTX PRO Blackwell servers) and NVIDIA messaging emphasizes NVFP4/quantized paths for better price‑performance. Plain English: vendors are competing on efficiency, not just raw speed. (investor.nvidia.com)
  • Telco‑cloud buildout: Verizon and AWS announced new long‑haul fiber to cut latency between AWS data centers, which can help multicloud AI costs by improving utilization. Plain English: faster pipes keep GPUs busier. (reuters.com)

What it means for your AI unit economics

  1. Use the new public GPU prices as leverage. Lock short‑to‑midterm commitments only when they beat expected near‑term market softening; otherwise ride on‑demand and diversify vendors. (lambda.ai)
  2. Expect incremental $/token improvements from software alone. Pilot TRT‑LLM 0.21.0 features (quantization, chunked attention, prompt‑lookup speculative decoding) on your heaviest models. (nvidia.github.io)
  3. Track power‑access milestones. Faster interconnections reduce project risk and time‑to‑serve, which shows up in lower cloud quotes and better tokens per watt as cleaner grid power replaces stopgap generation. (gibsondunn.com)
  4. Watch platform line items. New Agent/Agent Engine runtime charges can outweigh model price cuts for agentic apps—budget and route workload accordingly. (docs.cloud.google.com)

If you want, next week’s brief can add a tiny “dash” panel that tracks: posted B200/GB200/GB300 rental rates by provider, interconnection rulemaking milestones, and runtime/library releases that move tokens/sec or watts/token.