Why does Paimon recommend keeping bucket data size between 200MB and 1GB?

Bucket count directly caps write parallelism and shapes file sizes, so it sits on a tradeoff between two failure modes. Too many buckets for the actual data volume means each bucket ends up holding very little data, which produces lots of small files — more files for readers to open and merge, and more filesystem/object-store metadata overhead per byte of actual data.

Too few buckets for the data volume has the opposite problem: each bucket becomes a bottleneck, since a bucket can only be written and compacted by limited parallelism, and merges within an oversized bucket take longer and touch more data per compaction. The 200MB–1GB range is the practical middle ground the Paimon community has found balances per-file overhead against per-bucket write/compaction throughput for typical workloads.

In practice, this means periodically checking actual per-bucket size (via the $files system table) against expected data growth, and adjusting the bucket count — or switching to Dynamic Bucket — as a table's volume changes over its lifetime rather than treating the initial bucket count as permanent.