A lot of people are probably struggling with LLM inference costs, but lately, a technique called speculative sampling has been getting attention.

Here’s how it works: a smaller model predicts the results first, and then a larger target model verifies them all at once using GPU parallel processing. This can reduce the number of target model calls by more than five times, dramatically lowering inference costs.

Think of it as the draft model quickly creating a rough draft, while the main model efficiently verifies it. The key point is that you can save computing resources while maintaining the same output quality.