Coordinator cannot load mysql and postgres extensions for sql ingest
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
0.21.1
but everything up to master seems affected
### Description
mysql and postgres sql ingest require the corresponding metadata extensions to be loaded, which results in an error because both modules are extenting SQLMetadataStorageDruidModule and calling its configure function which includes:
JsonConfigProvider.bind(binder, "druid.audit.manager", SQLAuditManagerConfig.class);
Since this bind can only be called once, the second metadata extension loaded encounters an exception at this point and fails to load, which in turn makes the sql ingestion / firehose module of that extension not load.
### Steps to reproduce
put
druid.extensions.loadList=["postgres-metadata-storage","mysql-metadata-storage"]
in your config and start coordinator.
### Stacktrace
Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:
coordinator |
coordinator | 1) A binding to com.google.common.base.Supplier was already configured at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.metadata.storage.postgresql.PostgreSQLMetadataStorageModule).
coordinator | at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.metadata.storage.mysql.MySQLMetadataStorageModule)
coordinator |
coordinator | 2) A binding to org.apache.druid.server.audit.SQLAuditManagerConfig was already configured at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:152) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.metadata.storage.postgresql.PostgreSQLMetadataStorageModule).
coordinator | at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:152) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.metadata.storage.mysql.MySQLMetadataStorageModule)
Contributor guide
Research direction
Start with SQLMetadataStorageDruidModule.configure and the MySQLMetadataStorageModule and PostgreSQLMetadataStorageModule mentioned in the stack trace. Reproduce the failure with both entries in druid.extensions.loadList, then inspect how the shared SQL audit configuration is bound when both extensions load. Done means the coordinator starts with both metadata extensions enabled and their SQL ingest modules available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql, postgresql
- Domain
- backend, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100