Explain the internal working of segment merging and merge policies?

Merging physically rewrites several existing segments into one new, larger segment: postings lists are combined, deleted documents are dropped entirely, and shared structures like the term dictionary are rebuilt for the merged set. The old segments are only deleted once the new one is fully written and the commit referencing it succeeds - so a crash mid-merge doesn't corrupt the index.

TieredMergePolicy, the default, doesn't merge strictly in creation order. Instead it groups segments into tiers by size and picks merge candidates that minimize a cost estimate factoring in both segment size and the proportion of deleted documents each segment carries - segments with more deletions are prioritized since merging them reclaims more wasted space per unit of merge work.

The ConcurrentMergeScheduler executes selected merges on background threads, with configurable I/O throttling so merge activity doesn't starve foreground indexing and search of disk bandwidth. Because merging is a write-amplifying process - the same data gets rewritten multiple times over an index's life as small segments repeatedly combine into larger ones - the policy's tiering thresholds are effectively a knob balancing write amplification against query-time segment fan-out.