The short answer
Retrieval quality in an AI search product is bounded by two things: how good the embedding model is, and how cheaply you can run it across an index. The first is a research question, the second an engineering one — and it is usually the second that keeps the product standing.
Perplexity's engineering team has published the second side in detail: the serving infrastructure behind pplx-embed and the ranking models used across its search, Computer and API platform. This piece draws out the general lessons from that stack.
What is an embedding?
An embedding turns a piece of text into a sequence of numbers. The aim is to preserve meaning: two sentences that mean similar things end up with similar number sequences. That is how search works — queries and documents land in the same space and the nearest neighbours get retrieved. This is what separates it from keyword matching: a query for "car rental" can find a page about "auto hire" that never contains those words.
The critical detail here is model size. Embedding models are small Transformers, usually under a billion parameters. Being small keeps costs down but creates a different problem: the model is so small that the work around running it can cost more than the model itself.
Two kinds of traffic, one engine
Perplexity frames embedding serving as two workloads, and the difference between them determines everything:
| Workload | When | What it optimises |
|---|---|---|
| Batch embedding | Building or re-indexing the vector database | Throughput — minimising cost |
| Online embedding | At query time, a single short text | Latency — answering fast |
| Scoring | Ranking documents after vector search | A balance of both |
The team's most notable decision: they did not build a separate embedding engine. The reasoning is technical and elegant — batch embedding resembles compute-bound prefill, while online embedding, often only a few tokens, resembles memory-bound decode. The kernels in their LLM stack already had the right shape, so they were reused.
Three services: Ivy, Tulip and ROSE
Three services handle a request, and the split is not incidental:
- Ivy — an HTTP gateway written in Rust. It does the CPU-side work: JSON parsing, tokenisation, input templating, batch splitting. It also chops large-batch requests into chunks and load-balances them across replicas.
- Tulip — the inference server interface. A gRPC server built with Rust, tokio and tonic; it handles scheduling and batching before dispatching to the engine.
- ROSE — the inference engine itself. Mostly Python; it holds the kernels, layers and model definitions, and manages CUDA graphs.
The pattern generalises: CPU work and GPU work are split into separate services, and the CPU side is written in a systems language. On small models, tokenisation and JSON parsing are expensive enough to leave the GPU waiting, which is why they did not stay in Python.
Why is the scheduler deliberately simple?
Tulip takes requests first-come, first-served — there is no elaborate priority scheme. That is not laziness but a decision grounded in measurement.
At the sequence lengths Perplexity serves, and for small embedding models, the linear cost of dense layers dominates the quadratic cost of attention. The consequence: latency is roughly proportional to token count, not sequence count. Once a batch saturates the GPU — around 512 tokens on a sub-billion-parameter model — packing in more sequences does not improve efficiency.
The practical rule that follows: do not optimise a scheduler before measuring your own stack. Where the quadratic term dominates, clever scheduling pays; where the linear term dominates, the same effort is wasted.
The real bottleneck: kernel launching
On small batches something surprising happens: the CPU cost of launching GPU kernels can exceed the GPU's actual compute time. The GPU finishes and waits for the next instruction.
The fix is CUDA graphs. The whole model is captured into a single graph so every launch collapses into one driver call. Because embedding models are small, the point where GPU work exceeds launch cost arrives late: thousands of tokens and tens of sequences.
But graphs have a price: each configuration has to be captured separately. Even with token counts padded to multiples of 64 or 256, that still yields thousands of graphs and minutes of capture per model. Perplexity's answer is lazy capture: each configuration runs eagerly once, then gets captured on its second hit. This gives up p99 latency at startup but spreads minutes of work across hours.
Not waiting on the wait
The second mechanism is the LazyTensor. Normally an inference step blocks until the GPU finishes, and the CPU idles through it. Instead, a LazyTensor tracks a page-locked host buffer, an asynchronous copy and a CUDA event; the step function returns immediately rather than blocking.
The result: a Rust task waits on batch N while the CPU starts preparing N+1. The gain comes not from new hardware but from filling idle time.
Kernel selection is still done by hand
ROSE supports several attention backends for ragged inputs: FlashInfer 2, FlashInfer 3 and FlashAttention 4. The team reports FlashAttention 4 is generally faster, but FlashInfer 3 beats it on Qwen-based models at very long sequence lengths — so the choice is made case by case.
One more detail: when serving an embedding model, ROSE does not instantiate a KV cache at all and dispatches to ragged attention variants to avoid padding. In embedding there is no next token to produce; that inherited piece of LLM machinery would only burn memory here.
Why is this so hard?
Counter-intuitively, the difficulty of embedding serving comes not from the model's size but from its smallness. In a large language model the GPU does so much work that everything around it disappears into the noise: a few milliseconds of tokenisation are invisible next to seconds of generation. With a small embedding model the situation inverts — the model finishes in microseconds and every delay around it lands directly on the bill.
Scale sits on top of that. In a search product, embedding runs twice: once over billions of documents while the index is built, and once on every query. In the first, a single percentage point of efficiency shows up directly in the hardware bill; in the second, a single millisecond is added directly to what the user waits. The same model has two economies, and they want optimising in opposite directions.
What was the measurement made against?
The most easily skipped part of an engineering write-up is how the comparison was set up, yet that is exactly what determines how meaningful the result is.
Perplexity benchmarks against vLLM 0.22.0 in BF16, on real model weights and real eval-derived inputs. Warmup runs verify that cosine similarity diverges by no more than 0.1 percent — meaning the speed gain was not bought by corrupting the output. Four suites are measured: low-latency embeddings (batch 1; 128, 512 and 4,096 tokens), low-latency scoring (batches of 5, 25 and 50 at 512 tokens), high-throughput embeddings (batch 100 across four concurrent processes) and high concurrency (1 to 16 concurrent requests, including Ivy's tokenisation and network overhead).
That last parenthesis matters: including network and tokenisation overhead makes the figure a production number rather than a lab one. A benchmark measuring only GPU time would have hidden exactly the problems this write-up describes.
What to take from this
Transferable lessons for anyone building their own retrieval stack:
- Measure where the time actually goes first. On small models the bottleneck is often not the model but the launching and data preparation around it.
- Ask about shape before building a separate engine. If batch embedding resembles prefill and a single query resembles decode, your existing kernels already fit.
- Batch by token count, not sequence count. Past the saturation point, adding sequences buys nothing.
- Do not underestimate the CPU side. Tokenisation and parsing can leave the GPU waiting; that work can move to a systems language.
- Benchmark on your own data. Perplexity verifies its comparisons on real weights and eval-derived inputs, holding cosine similarity divergence within 0.1 percent.
In summary
The summary of Perplexity's account is this: on the GPU side, embedding inference has largely converged across engines on mature hardware — the gap between engines has closed. The wins now sit in the harness around the model: CUDA graph management, asynchronous result tracking and a fast request path.
That is a generalisable observation for the field. Choosing the model is the easy part; what keeps a product standing is collecting the wasted milliseconds while running that model millions of times a day.
One caveat to close on: the numbers and choices here come from Perplexity's own measurements, calibrated to its own workload. If your sequence lengths, model size and hardware differ, so will the saturation point and the right scheduler. What transfers is not the figures but the questions asked.