Explain how hidden partitioning differs from Hive-style partitioning?

Hive-style partitioning requires a physically separate partition column (often derived and stored by an ETL job ahead of time, like event_date) baked directly into the directory structure, and users must explicitly filter on that partition column in their queries to get efficient pruning.

-- Hive-style: user must know and filter on the derived partition column
SELECT * FROM events WHERE event_date = '2026-01-15';

-- Iceberg hidden partitioning: user filters the natural column directly
SELECT * FROM events WHERE event_time >= '2026-01-15 00:00:00';

Iceberg's hidden partitioning instead computes the partition value automatically from a natural data column (like deriving a day-level partition from a event_time timestamp) and tracks that transform as part of the table's metadata, so the query planner can translate a filter on the natural column into correct partition pruning without the user ever needing to know a separate partition column exists.

The practical consequence is significant: in Hive, a user who filters on event_time instead of the "correct" partition column event_date silently gets a full table scan with no error or warning, while in Iceberg, filtering on the natural column is the expected, normal way to query the table, and pruning happens correctly regardless of exactly how the filter is phrased, as long as it constrains the underlying source column the partition transform is based on.