How does Lucene score documents (TF-IDF vs BM25)?

Lucene ranks matching documents using a Similarity implementation, and since Lucene 6 the default has been BM25Similarity, replacing the older classic TF-IDF-based vector space model.

Both approaches reward documents where a query term appears frequently (term frequency) and penalize terms that are common across the whole corpus (inverse document frequency). The key difference is how they treat repeated occurrences and document length:

• Classic TF-IDF lets term frequency keep contributing to the score roughly proportionally, with limited saturation.
• BM25 introduces a saturation parameter (k1) so extra repetitions of a term add diminishing returns, plus a length-normalization parameter (b) that controls how much a document's length relative to the average penalizes its score.

In practice, BM25's saturation curve tends to produce more intuitive rankings - a document mentioning a term 50 times isn't scored wildly higher than one mentioning it 5 times - which is why it's been the sensible default for years.