AI / LlamaIndex Interview Questions
What is the difference between refine and compact response modes?
Both refine and compact build an answer by working through the retrieved Nodes with the LLM, but they differ in how many calls that takes.
refine processes Nodes one at a time: it generates an initial answer from the first Node, then feeds each subsequent Node to the LLM along with the current answer, asking it to refine or confirm the answer. With k retrieved Nodes, that's up to k sequential LLM calls.
compact instead concatenates as many Node texts as fit into a single prompt within the LLM's context window, only moving to a second call if the Nodes don't all fit. This usually means far fewer calls than refine for the same Nodes, which is why compact is the default and generally faster and cheaper, while refine can sometimes produce a more carefully considered answer at higher cost.
More Related questions...
