Faster search with a smaller model
We replaced a slow production search path with a smaller model trained for the client’s workload.
Claude Sonnet delivered strong relevance, but responses were taking 4 to 5 seconds in production. That was too slow for the search experience and expensive to operate at scale.
We built an evaluation set from the client’s search traffic, trained a smaller model for that domain, compared relevance against Claude Sonnet, and designed a serving and fallback path for production.
- Relevance comparable to Claude Sonnet on the client’s evaluation set
- Latency reduced from 4 to 5 seconds to about 500 milliseconds
- Search made 8 to 10 times faster
- Approximately $250,000 in annual infrastructure savings