Search latency reduced from seconds to milliseconds
A production search workload was taking 4 to 5 seconds. We trained a smaller model that delivered relevance comparable to Claude Sonnet, reduced latency to about 500 milliseconds, and saved the client approximately $250,000 a year in infrastructure costs.