How can you optimize indexing throughput for large-scale data?

Indexing throughput tuning trades some memory, durability latency, or feature richness for raw speed, so each lever below should be chosen deliberately rather than applied blindly:

1. Increase the RAM buffer size (IndexWriterConfig.setRAMBufferSizeMB) so more documents accumulate before a flush, reducing the number of small segments created.
2. Avoid storing fields you don't need to display later - stored fields and term vectors both add write cost with no indexing benefit.
3. Batch documents and commit infrequently rather than after every add, since commits are relatively expensive fsync operations.
4. Use multiple indexing threads feeding a single shared IndexWriter, which is thread-safe for concurrent document additions.
5. Disable norms and omit term frequencies/positions on fields that don't need relevance scoring or phrase queries.
6. Increase the merge scheduler's I/O throttle cautiously, since faster merges reduce backlog but compete more aggressively with concurrent search traffic.

Measuring documents-per-second before and after each change is important, since some of these settings interact - a larger RAM buffer, for instance, also means more work lost if the process crashes before a commit.