LLM Savings Surge: Cutting Costs This Week!
The LLM Cost‑Cutter
Weekly briefing on advances that shrink the cost of large language models (Oct 28–Nov 4, 2025)
TL;DR: This week’s real cost movers are (a) fresh, public GPU-rental prices for Blackwell that keep drifting down, (b) a big Microsoft capacity deal that should ease scarcity, (c) U.S. regulators formally pushing to speed up data‑center grid hookups, and (d) software/runtime updates that squeeze more tokens per watt. One notable counter‑trend: new runtime charges from cloud AI platforms in certain regions.
1) Public GPU prices: Blackwell competition intensifies
- Lambda’s live page lists 8×B200 at $4.99 per GPU‑hour on demand, with multi‑week cluster terms “as low as” $3.79/GPU‑hour; H100 shows $2.99/GPU‑hour. This page was updated this week. Plain English: market‑rate reference points for negotiating and planning just got clearer. (lambda.ai)
- Runpod shows B200 available on secure cloud at $5.69/hr, plus new per‑second serverless rates: B200 at $0.00190–$0.00240/sec (active vs. flex). Plain English: more providers, more transparent pricing, finer‑grained billing. (runpod.io)
- CoreWeave’s catalog (updated last week) highlights B200 and new Blackwell RTX PRO 6000 server editions, signaling broader Blackwell availability across instance types. Plain English: more SKUs means more ways to right‑size spend. (github.com)
Why this matters for costs and intelligence per watt
- More posted prices + more supply generally push $/GPU‑hour down. Pair these with better runtimes (below) and you get fewer dollars and watts per useful token. (lambda.ai)
2) Big capacity: Microsoft signs $9.7B, five‑year deal for GB300 racks
- Microsoft inked a $9.7B contract with IREN for Nvidia GB300 systems, deploying ~200 MW of liquid‑cooled capacity at IREN’s Childress (TX) campus through 2026. Plain English: more cutting‑edge inference/training gear will come online without Microsoft building all of it itself. (reuters.com)
Why this matters for costs and intelligence per watt
- More near‑term capacity reduces scarcity premiums on GPU time; GB300 also claims sizable per‑watt performance gains versus GB200, improving tokens per watt as it ramps. (tomshardware.com)
3) Policy tailwind: DOE pushes FERC to fast‑track large‑load interconnections
- On Oct 23 (covered over the past week), the U.S. Energy Secretary asked FERC to initiate a rulemaking to standardize and speed grid hookups for big loads like AI data centers, with a draft ANOPR and an April 30, 2026 response date proposed. Plain English: less waiting for power means data centers come online sooner and cheaper. (gibsondunn.com)
- Related: PJM (the largest U.S. grid operator) published progress updates and continues queue reforms to shorten interconnection timelines. Plain English: the biggest queue is clearing faster, which lowers project risk and cost. (insidelines.pjm.com)
Why this matters for costs and intelligence per watt
- Faster, more predictable power access lowers financing and temporary‑generator costs. Over time it shifts compute to cleaner, cheaper grid power—improving $/kWh and watts per token. (gibsondunn.com)
4) Runtime and kernel upgrades: more tokens on the same watts
- NVIDIA’s TensorRT‑LLM 0.21.0 adds new quantization paths (w4a8 MXFP4/FP8), chunked/sliding‑window attention, prompt‑lookup speculative decoding, and KV‑cache transfer improvements—aimed squarely at higher throughput and lower latency on Hopper/Blackwell. Plain English: the same GPUs output more useful tokens per unit time and energy. (nvidia.github.io)
- PyTorch 2.9 shipped recently with performance and hardware enablement improvements (including Arm), reducing overhead in compiled graphs and decoding paths. Plain English: the common framework many teams use just got a bit faster and broader. (pytorch.org)
- vLLM’s public Q4 roadmap (updated six days ago) focuses on cutting CPU overhead and improving frontier‑cluster serving—areas that often bottleneck throughput/$ at scale. Plain English: serving stacks keep maturing for big clusters. (github.com)
Why this matters for costs and intelligence per watt
- These optimizations increase tokens/sec without new hardware, multiplying savings when combined with cheaper GPU‑hours and better batching/scheduling. (nvidia.github.io)
5) Fresh research: faster “thinking” without extra watts
- SpecDiff‑2 (Nov 1) proposes a diffusion‑based drafter aligned to an autoregressive verifier; reports up to +55% tokens/sec over strong baselines and up to 5.5× over standard decoding at no accuracy loss (lab results). Plain English: better speculative decoding = same answers, fewer milliseconds and joules. (arxiv.org)
Why this matters for costs and intelligence per watt
- As these methods land in mainstream servers (TRT‑LLM, vLLM, etc.), you get cheaper, faster generation per request, compounding with hardware and price drops. (nvidia.github.io)
6) Platform pricing changes you should know
- OpenAI: AgentKit/Agent Builder billing began Nov 1, 2025 (storage metering and other platform line items). Plain English: some “free” prototyping surfaces now meter usage—plan budgets accordingly. (openai.com)
- Google Cloud: Vertex AI Agent Engine runtime will start charging in more regions on Nov 6 (Singapore, Melbourne, London, Frankfurt, Netherlands). Plain English: new line items can raise total cost of ownership for “agent” apps even if raw model token rates are unchanged. (docs.cloud.google.com)
Why this matters for costs and intelligence per watt
- API bills are the sum of model tokens and platform runtime add‑ons. Keeping an eye on these fees—and shifting workloads to cheaper runtimes when possible—directly lowers $/request. (cloud.google.com)
7) Quick market signals (context for the next few weeks)
- More Blackwell SKUs are surfacing (e.g., RTX PRO Blackwell servers) and NVIDIA messaging emphasizes NVFP4/quantized paths for better price‑performance. Plain English: vendors are competing on efficiency, not just raw speed. (investor.nvidia.com)
- Telco‑cloud buildout: Verizon and AWS announced new long‑haul fiber to cut latency between AWS data centers, which can help multicloud AI costs by improving utilization. Plain English: faster pipes keep GPUs busier. (reuters.com)
What it means for your AI unit economics
- Use the new public GPU prices as leverage. Lock short‑to‑midterm commitments only when they beat expected near‑term market softening; otherwise ride on‑demand and diversify vendors. (lambda.ai)
- Expect incremental $/token improvements from software alone. Pilot TRT‑LLM 0.21.0 features (quantization, chunked attention, prompt‑lookup speculative decoding) on your heaviest models. (nvidia.github.io)
- Track power‑access milestones. Faster interconnections reduce project risk and time‑to‑serve, which shows up in lower cloud quotes and better tokens per watt as cleaner grid power replaces stopgap generation. (gibsondunn.com)
- Watch platform line items. New Agent/Agent Engine runtime charges can outweigh model price cuts for agentic apps—budget and route workload accordingly. (docs.cloud.google.com)
If you want, next week’s brief can add a tiny “dash” panel that tracks: posted B200/GB200/GB300 rental rates by provider, interconnection rulemaking milestones, and runtime/library releases that move tokens/sec or watts/token.