Including TableProvider IO dependencies as `provided` can lead to ClassDefNotFoundErrors for users
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
The SQL extension uses ServiceLoader to load table providers: https://github.com/apache/beam/blob/v2.22.0/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchemaFactory.java#L85
And the IOs used in the built-in table providers (e.g. pubsub json, avro, parquet, ...) are included as `provided` dependencies, not `compile`. This means that we are able to build the sql extension jar, and AutoService picks them up and records them in META-INF/services. However because they are `provided` dependencies, the necessary IO modules are not marked as dependencies for the SQL extension.
This means consumers of :sdks:java:extensions:sql (like :sdks:java:extensions:sql:jdbc) can end up with ServiceLoader registering a TableProvider that will fail with ClassDefNotFoundError, unless they add the necessary dependency.
Possible solution: Refactor TableProvider to beam core (or another shared module) in order to solve Service Loader errors with SchemaCapableIOTableProviderWrapper. Remove getTableType overrides of SchemaCapableIOProviderWrapper when fixed.
Imported from Jira [BEAM-10516](https://issues.apache.org/jira/browse/BEAM-10516). Original Jira may contain additional context.
Reported by: slukas.
Contributor guide
Assessment
This issue has not been assessed yet.