Which is better and why: the classic consumer rebalance protocol or the new KIP-848 protocol?

The classic protocol (used by default for years) is eager: every rebalance revokes all partitions from all members first, then reassigns everything from scratch, which means every consumer briefly stops processing, even partitions it ends up keeping. The KIP-848 next-generation protocol, generally available since Kafka 4.0, moves partition assignment logic to the group coordinator (rather than a client-side leader) and only reassigns the specific partitions that actually need to move, letting members keep processing unaffected partitions throughout.

For most current deployments, KIP-848 is the better choice: it meaningfully reduces the processing pause during rebalances, is more resilient to slow or buggy client-side assignment logic since the server now owns that decision, and is where new Kafka development is focused going forward. The classic protocol still has a place in environments running older client libraries not yet updated to support the new protocol, or in scenarios needing a custom client-side assignment strategy that hasn't been ported to the new coordinator-driven model yet.

The practical guidance: default to the new protocol on Kafka 4.0+ clusters with up-to-date clients, and only stay on the classic protocol where a specific compatibility constraint requires it.