BigData / Apache StreamPark Interview questions
Explain the internal working of StreamPark's multi-version Flink support through custom classloading?
Running several Flink versions from one long-lived Console process is trickier than it sounds, because different Flink versions ship different, sometimes incompatible, classes on their client-side classpaths. If Console naively loaded every registered Flink version's classes into the same JVM classpath, version collisions would be inevitable.
StreamPark works around this with a custom classloader per registered Flink distribution, using a child-first (parent-last) loading strategy: for classes belonging to a specific Flink version's libraries, the classloader checks its own version-specific classpath before delegating up to the shared parent classloader. This is the inverse of the JVM's normal parent-first delegation, and it's exactly what's needed to keep, say, Flink 1.17 and Flink 1.20 client classes from colliding when both are registered on the same Console instance.
The practical effect: when a job configured for Flink 1.17 is submitted, Console resolves classes through that version's classloader; a job configured for Flink 1.20 resolves through a separate one. Both can be in flight from the same running Console process, on the same cluster type (YARN or Kubernetes), without one version's client libraries leaking into the other's classpath.
More Related questions...