Large language models are stateless. Ask the same question twice and you pay twice. Semantic caching remembers the meaning of previous prompts and serves cached responses when similarity exceeds a configurable threshold.
How it works
When a request arrives at the CIPHER gateway, we compute an embedding of the prompt, compare it against an in-memory vector index, and return a cached response if the match is strong enough. If not, the request proceeds to the model and the result is stored.
Real results
In pilot deployments with customer support and documentation Q&A workloads, we have seen cache hit rates between 25% and 45%, directly reducing token spend by a similar proportion.
The cache is fully configurable: you set the similarity threshold, TTL, and whether to cache per tenant or globally.