LLM Savings: Your Weekly Cost-Cutting Update!

The LLM Cost‑Cutter

Weekly briefing on advances that shrink the cost of large language models (Feb 18–Feb 24, 2026)

Pricing announcements from AI labs this week

  • OpenAI: No new per‑token API price cuts announced this week; the company completed retirement of GPT‑4o in ChatGPT on February 13, 2026, with petitions and press coverage since then. API access to some 4o variants remains for now, per deprecation notes. No pricing changes were announced alongside these model retirements. (businessinsider.com)
  • Anthropic: Rolled out Claude Sonnet 4.6 as the default in its apps without announcing API price changes; consumer plans remain Pro ($17/month annual; $20 monthly) and Max ($100/month). (timesofindia.indiatimes.com)
  • Google: No new per‑token price moves this week; Vertex AI’s Agent Engine continues to bill runtime and add‑ons as of Jan 28, 2026. (cloud.google.com)
  • Mistral: No new public API price changes posted this week. (mistral.ai)

Bottom line: No headline API price cuts this week; cost progress is coming from hardware, power, and serving‑stack efficiencies.

—

1) Research: Faster, cheaper decoding that preserves quality

  • What’s new

    • Privacy‑aware split inference over a wide‑area network combines “lookahead” decoding with speculation so a small local GPU does the sensitive parts while a cloud GPU handles the rest, keeping throughput high. Reported 8–9 tokens/sec over ~80 ms WAN on 7B–12B models, with projections of 15–19 tok/s at 20 ms RTT. Plain English: keep private data local, send only intermediate math to the cloud, and still go fast. (arxiv.org)
    • Dynamic Delayed Tree Expansion improves multi‑path speculative decoding by delaying branching and using a learned selector; authors report about 5% higher average throughput than strong baselines across models. Plain English: draft smarter, not just more, so more of the drafted text gets accepted. (arxiv.org)
    • MoE‑Spec caps the number of experts loaded during verification when serving Mixture‑of‑Experts (MoE) models, yielding 10–30% higher throughput at similar quality. Plain English: load fewer “specialists” at once so memory traffic drops and tokens/sec rise. (arxiv.org)
  • Why it matters for LLM costs

    • These methods reduce time per token without bigger GPUs. That cuts dollars per million tokens (since you pay for GPU time) and boosts “intelligence per watt” by wasting fewer cycles on rejected drafts or unnecessary memory movement. Over time, they stack with hardware gains.

—

2) Hardware and supply: Memory outlook and Nvidia catalyst

  • What’s new

    • Investors expect more detail on Blackwell/Rubin inference‑per‑watt in Nvidia’s Feb 25 earnings; sentiment this week also improved on signs SK Group will boost AI memory output, easing HBM supply constraints. Plain English: more HBM supply tends to lower GPU system cost or improve availability. (barrons.com)
    • Analysts and trade press have been flagging the risk of HBM price competition in 2026 as Samsung ramps, which would shift pricing power toward chip customers like Nvidia. Plain English: if HBM gets cheaper or more available, accelerators and total cost per token trend down. (digitimes.com)
  • Why it matters for LLM costs

    • HBM is a major bill of materials item for AI GPUs. More suppliers and capacity can trim accelerator pricing and reduce queue times, improving both absolute cost and throughput per megawatt (you can actually get and fully load the boxes).

—

3) Power and policy: Off‑grid buildouts and utility deals

  • What’s new

    • The “shadow power grid” trend accelerated: large AI campuses are securing private gas‑fired generation to avoid grid interconnect delays. Plain English: self‑power to get online faster, but with carbon trade‑offs. (washingtonpost.com)
    • Utility‑backed, long‑dated power deals continue to set precedents (e.g., 1.4 GW for Oracle’s Michigan campus through February 2045), keeping costs predictable and avoiding cost shifts to other ratepayers. Plain English: lock in firm megawatts for decades so your GPU fleet isn’t idling. (barrons.com)
    • Advocacy pushback on special data‑center tariffs intensified in Wisconsin, arguing against rate hikes on households. Plain English: more scrutiny on who pays for grid upgrades; expect stricter conditions but clearer tariffs over time. (naacp.org)
  • Why it matters for LLM costs

    • Predictable, dedicated power reduces downtime and price spikes, lowering dollars per million tokens. Faster hookups and co‑location options shorten time‑to‑revenue for new capacity. Policy clarity also de‑risks projects, reducing financing costs.

—

4) Facilities and cooling: Toward lower PUE at higher density

  • What’s new

    • Market analysts see rapid growth for liquid cooling as AI racks densify, pushing direct‑to‑chip and immersion mainstream. Plain English: more heat per rack means you need liquids, not just air, to keep GPUs efficient. (globenewswire.com)
    • Microsoft continues its rollout of “zero‑water” or recycled‑water cooling designs in new builds (Arizona, Wisconsin, Iowa), trading a small energy increase for major water savings. Plain English: water use drops to near‑zero while maintaining efficiency with modern chillers. (microsoft.com)
  • Why it matters for LLM costs

    • Better cooling pushes PUE closer to 1.0 and supports denser, more efficient GPU racks. That raises tokens/sec per square foot and per watt, trimming the energy line in your cost per million tokens.

—

5) Cloud GPU “street prices” you can use in negotiations

  • What’s new

    • Runpod posted serverless per‑second rates that include high‑end parts:
      • B200 serverless “active” workers shown at $4.46/hour (and $5.58/hour flex), with H200 at $3.35/hour active; on standard pods, public pages list multiple SKUs but some on‑demand line items require contacting sales. Plain English: public cards help you benchmark quotes and find sweet spots for inference fleets. (runpod.io)
    • CoreWeave continues to publish reference rates for multi‑GPU nodes (e.g., HGX H100/H200 and B200 configurations), aiding price discovery and negotiation even if your final quote differs. Plain English: you can anchor vendor talks to public lists. (coreweave.com)
  • Why it matters for LLM costs

    • Transparent hourly pricing plus smarter serving (speculative decoding, KV‑compression) lets you hit lower effective $/M tokens by matching job size and SLA to the cheapest viable silicon.

—

6) Anthropic platform moves without token‑price changes

  • What’s new

    • Anthropic showcased new Claude capabilities and integrations and promoted consumer‑tier options, while third‑party coverage highlighted Sonnet 4.6 replacing prior defaults on the user side—with no API per‑token cuts announced. Plain English: better capability/UX at the same token prices. (barrons.com)
  • Why it matters for LLM costs

    • If quality rises at constant price, you get more useful work per token. Paired with serving optimizations, “intelligence per watt” and per dollar still improve even when list prices don’t move.

—

Quick takes

  • Nvidia earnings (Feb 25) could bring firmer throughput‑per‑MW disclosures for Blackwell/Rubin—useful for planning 2026–27 inference costs and migration timing. (barrons.com)
  • Private power deals (e.g., 500 MW rental for an AI customer starting 2027) show a pipeline of off‑grid capacity that may stabilize compute availability but warrants carbon accounting. (datacenterdynamics.com)

—

This week’s impact in plain English

  • No new per‑token API price cuts, but total cost per answer keeps sliding as: decoding gets smarter (higher token acceptance), HBM supply loosens, long‑term power gets locked in, and cooling supports denser racks. Together, these boost tokens per dollar and “intelligence per watt.”

—

Action checklist (1–2 hour tasks)

  1. Pilot a speculative‑decoding update: try delayed branching or MoE expert budgeting on one production route; measure tokens/sec, GPU memory, and acceptance rate. Target ≥10% throughput gain. (arxiv.org)
  2. Re‑shop inference fleets using posted B200/H200 serverless rates as anchors; ask vendors to quote committed discounts for always‑on workers and compare to your current $/M tokens. (runpod.io)
  3. If you host in PJM/Midwest, map exposure to power‑tariff changes and explore self‑generation or long‑dated PPAs to cap volatility. Use recent Oracle/DTE terms as a sanity check for “no cross‑subsidy” structures. (barrons.com)
  4. Start a liquid‑cooling ROI sketch: estimate PUE delta vs. air, feasible rack density, and floor space savings for a 2026–27 refresh. (globenewswire.com)
  5. For privacy‑sensitive use cases, test split inference with lookahead on a small model to see if you can keep data local without losing speed. (arxiv.org)

—
Sources: OpenAI model retirement notes and coverage; Anthropic pricing and product updates; Google Vertex AI pricing (runtime and add‑ons); Nvidia/HBM supply reporting; Runpod and CoreWeave pricing pages; policy and power developments; and peer‑reviewed/archival research on speculative decoding and split inference. (help.openai.com)