Usage of "private" Spark APIs

Frameless is depending on portions of Spark for which there is no binary compatibility commitment. For example, Frameless uses StaticInvoke, which is part of the org.apache.spark.sql.catalyst.expressions.objects package. If you look at the (bountiful) mima exclusions in Spark, the entire org.apache.spark.sql.catalyst package is not checked for binary compatibility.

I don't really consider this a bug with frameless, but I wanted to at least raise it as a concern as it recently bit us at work.

backstory for those who care

At work we use Databricks runtime 3.5. Databricks claims that this runtime uses Spark 2.2. However, we ran into a bewildering issue with a binary incompatibility between Frameless and the runtime Spark version (related to org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke). After quite a bit of investigation, we realized that the Databricks runtime doesn't actually include Spark 2.2 proper, but a private fork of it that has some incompatible changes. It has a backported change from Spark 2.3 that is incompatible with Spark 2.2 (and the version of Frameless that is built against Spark 2.2). We can work around this particular issue by moving to Spark 2.3 and the Databricks 4.0 runtime, but it's tough to know what other incompatibilities could be lurking in the private forks, and I could envision other people running into similar issues (especially if they can't move to Spark 2.3).

Start by reviewing Frameless's use of Spark's org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke and the Spark 2.2 compatibility assumptions described here. Compare the reported Spark 2.2, Spark 2.3, and Databricks runtime behavior; the issue is resolved only when the project has a decided way to handle or document these private API incompatibilities.