Companies such as Fireworks and Together AI sell access to AI models developed by other labs. Many of those models have publicly downloadable weights: the numerical values learned during training. Customers could run the models themselves, yet they pay a provider to do it for them.
The video The Strange Economics of LLM Inference-as-a-Service explains why. Running a large model requires expensive hardware, and getting useful work out of that hardware takes considerable engineering. A provider has to answer requests quickly while keeping enough customers on its servers to cover the cost.
The business depends on how efficiently the provider runs the model and how consistently customers use it.
What an inference provider sells
Inference is the process of using a trained model. When someone sends a prompt to a language model, it processes the input and generates an answer in tokens, the pieces of text the model works with.
An inference service handles that process for customers. They send requests through an API, a software interface, instead of buying or renting GPUs and operating the model themselves.
The video separates the providers into four groups. Large cloud companies such as Amazon, Microsoft and Google offer inference alongside their other cloud services. GPU-focused cloud companies, including CoreWeave and Nebius, primarily rent computing capacity for training and inference.
Fireworks and Together AI focus more directly on serving models. They improve the software that runs them and charge customers for usage. Groq and Cerebras go further by developing different kinds of processors to run models faster.
Providers serving the same open-weight model compete with each other and with the lab’s own API. They need some advantage, such as lower prices, faster responses or more reliable availability.
Why charging for tokens may not cover the bill
The presenter uses a simplified DeepSeek V4 Pro example to show how difficult the economics can be.
Assume a server has eight Nvidia B200 GPUs, each rented for $6 an hour. The GPUs alone cost $48 an hour, before other computing resources, networking or staff.
The example uses prices of $1.74 per million input tokens and $3.48 per million output tokens. These are the prices cited in the video, rather than a guarantee of current rates. It also assumes customers send two input tokens for every output token the model generates.
Under those assumptions, producing a million output tokens earns $6.96: $3.48 for the output and another $3.48 for the associated input.
If the server generates 900 output tokens a second across its customers, it produces 3.24 million in an hour. That brings in about $22.55, less than half the GPU rental bill. It would need to generate roughly 2,000 output tokens a second continuously just to cover that bill.
This is an illustrative calculation, not a measured account of a provider’s business. Rental rates, actual performance and customers’ input-to-output ratios can all differ. But it explains why both usage and efficiency matter: the provider pays for the server even when customers are not keeping it busy.
Reading a prompt and writing an answer are different jobs
The video explains two stages of language-model inference: prefill and decode.
During prefill, the model processes the input prompt. Much of that work can happen in parallel, making it well suited to GPUs’ ability to perform many calculations at once.
During decode, the model generates the response step by step. Each new token depends on the text generated before it. The system repeatedly reads model data and information about the existing conversation to produce the next token.
This makes memory speed particularly important during decoding. A processor may be capable of more calculations than it can use because it is waiting for data. It also helps explain why providers often charge more for output tokens than input tokens.
Providers reduce the cost of these stages in several ways.
Reusing work and serving customers together
When a model processes a prompt, it creates a KV cache, stored information used by its attention mechanism to generate the response. If a later request begins with exactly the same text, the server may be able to reuse that information instead of processing the shared portion again.
Coding agents are a useful example. They repeatedly send instructions, tool descriptions, code and conversation history. Much of this text stays the same between requests, so caching can avoid substantial repeated work.
Reusing a cache still requires finding it, moving it to the right machine and processing any new text. It is generally cheaper than calculating everything again.
Batching improves efficiency in a different way. Rather than generating the next token for one customer, the server handles the next tokens for many customers together. The GPU can reuse model data across those requests.
With continuous batching, completed requests leave the group and new requests join it. Customers do not have to start and finish together.
This gives a busy provider an advantage: having more requests can make each request cheaper to serve. There are limits, however. Waiting to assemble a larger batch can delay responses, and more active requests need more cache memory.
Large providers can also assign prefill and decode to separate groups of GPUs. One group processes prompts, then sends the resulting cache to another group that generates the answers. The provider can give each stage the hardware it needs and prevent a long incoming prompt from delaying responses already being generated.
Changing how the GPU does the work
The presenter also explains custom kernels: small programs that tell a GPU how to carry out particular operations.
General-purpose software has to work across many models and hardware configurations. A specialized kernel can take advantage of the exact model, data format and GPU being used.
Some improvements come from moving data less often. For example, separate operations might each read data from memory, perform a calculation and write the result back. A fused kernel combines operations so that intermediate results can stay in faster memory.
Another technique, tiling, divides large calculations into smaller blocks. The GPU can reuse data within a block before fetching more.
FlashAttention uses this approach for attention calculations. It processes smaller blocks without storing the entire, potentially huge matrix of attention scores in the GPU’s main memory.
These optimizations take specialized engineering and do not work equally well everywhere. Software tuned for one GPU or workload may perform poorly on another.
The presenter argues that model developers can have a head start. They can prepare their serving software before releasing the weights and optimize it for their own customers’ requests. An outside provider then has to integrate the model and tune its own system.
What DeepSeek’s figures actually showed
The video cites DeepSeek’s February 2025 report on serving V3 and R1. DeepSeek reported about 73,700 input tokens a second during prefill or 14,800 output tokens a second during decode per eight-GPU H800 node.
Those figures describe its production workload. More than half of the input tokens were cache hits, meaning the system reused earlier calculations. The input figure therefore does not describe processing entirely new prompts at that speed.
There is also a correction to the video’s description of a 545% margin. DeepSeek calculated theoretical revenue by assuming all tokens were billed at R1 prices, then compared it with estimated GPU rental costs. The 545% figure was profit relative to that cost estimate, not an actual profit margin on sales.
DeepSeek explicitly said its real revenue was substantially lower. Its V3 prices were cheaper, its web and app services were free, and it offered nighttime discounts.
Building processors for inference
Groq and Cerebras try to improve performance by changing the hardware itself.
Groq’s Language Processing Unit uses a compiler to plan calculations and data transfers ahead of execution. Its design relies on fast on-chip memory and carefully scheduled movement of data.
Cerebras builds a processor covering almost an entire silicon wafer. Its design puts a large amount of computing and fast memory on one chip, reducing some of the need to move data between separate processors.
The video discusses their high output speeds for individual users. Those figures differ from a server’s combined output across many customers. A fast answer for one user does not, by itself, show how much work the whole system can perform or what each token costs to serve.
These providers also have to adapt their systems to new models. A model available on conventional GPUs may not immediately be supported on specialized hardware.
Why a model lab might use another company’s servers
The presenter’s closing argument is that building models and operating a global inference service may increasingly become separate businesses.
A lab needs computing capacity to train its next model. Serving a growing number of customers competes for those resources. Limited funding or access to advanced chips can make it harder to do both, and export restrictions can add further constraints for Chinese labs.
A lab may release a capable model without having enough infrastructure to answer requests quickly around the world. A separate provider can supply that infrastructure.
The presenter expects better open-weight models to increase demand for these services. That is a forecast. The opportunity described in the video is to make a model consistently available, respond quickly and serve enough requests efficiently to cover the cost of running it.