How do you troubleshoot excessive small files from streaming writes into a Paimon table?

Streaming writes commit frequently and in small batches, which naturally produces many small data files — left unmanaged, this degrades both read performance (more files to open and merge) and puts pressure on the underlying filesystem/object store.

1. Check the $files system table. Query my_table$files to see the actual file count and size distribution per partition/bucket, confirming small files really are the issue before tuning anything.
2. Verify compaction is running. Streaming writers can run compaction inline, but heavy-throughput pipelines are often better served by a dedicated, separately-scaled compaction job so writing isn't slowed down by compaction work.
3. Tune commit frequency. If the upstream job commits (checkpoints) too often relative to data volume, each commit produces disproportionately small files; widening the checkpoint interval reduces file count at the cost of end-to-end latency.
4. Review bucket count. Too many buckets for the actual data volume spreads writes thin, producing small files per bucket; Paimon recommends sizing buckets so each holds roughly 200MB–1GB, and reducing bucket count (or switching to Dynamic Bucket) if buckets are consistently under that.
5. Consider Postpone Bucket mode for very high-throughput, low-latency ingestion, deferring bucketing and merging to a scheduled batch compaction instead of doing it inline on every commit.