AI engineering
Before choosing a faster model, check the cache
Repeated documents are an opportunity to avoid repeated work. A small lesson from building SkipTheTerms.
Model selection gets a lot of attention. Call frequency deserves some too. If different people request the same transformation of an unchanged document, generating the same answer repeatedly may be unnecessary.
SkipTheTerms uses a hash of the document content to identify that opportunity. The backend looks for an existing summary before making a new request. A changed document produces a different hash, so an old summary is not automatically reused for new content.
A cache also needs a definition of what makes two requests equivalent. The prompt, model, language, and output format can affect the answer. In a growing system, those choices belong in the cache identity or its invalidation rules.
This is not a claim that caching solves every AI workload. Personalized or time-sensitive requests need different treatment. The point is to look for the repeated work your specific product creates.
My default question now is simple: does this request need a new generation? Sometimes the better engineering decision is to return work the system has already done.