Two layers that cut an LLM bill: prefix caching and semantic caching
In a production LLM application the cost piles up in two separate places. This guide separates prefix caching from semantic caching, and says what to measure.
Models
In a production LLM application the cost piles up in two separate places. This guide separates prefix caching from semantic caching, and says what to measure.