Explain the internal working of the manifest and commit protocol in Lance?

Every version of a Lance table is described by a manifest — a metadata file listing exactly which data files make up that version's state, along with schema information — and advancing to a new version means atomically committing a new manifest that supersedes the previous one.

The commit itself relies on the storage backend's own atomic, conditional-write capability (for example, S3's conditional PUT semantics) to guarantee that exactly one writer successfully advances the version from N to N+1 at a time, even if multiple writers attempt to commit concurrently — this is the mechanism behind LanceDB's optimistic concurrency control.

Because each manifest is a complete, self-contained description of a version's state (rather than a diff against the previous one), checking out an old version, or reading the latest version, both reduce to the same operation: read the relevant manifest, then read exactly the data files it references — there's no need to replay a sequence of incremental changes to reconstruct any particular version's state.