apache / apache/druid

Coordinator cannot load mysql and postgres extensions for sql ingest

Open
#11,733 2 comments 1 reaction 0 assignees View on GitHub
Area - Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

0.21.1
but everything up to master seems affected

### Description

mysql and postgres sql ingest require the corresponding metadata extensions to be loaded, which results in an error because both modules are extenting SQLMetadataStorageDruidModule and calling its configure function which includes:

JsonConfigProvider.bind(binder, "druid.audit.manager", SQLAuditManagerConfig.class);

Since this bind can only be called once, the second metadata extension loaded encounters an exception at this point and fails to load, which in turn makes the sql ingestion / firehose module of that extension not load.

### Steps to reproduce

put

druid.extensions.loadList=["postgres-metadata-storage","mysql-metadata-storage"]

in your config and start coordinator.

### Stacktrace

Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:
coordinator |
coordinator | 1) A binding to com.google.common.base.Supplier was already configured at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.metadata.storage.postgresql.PostgreSQLMetadataStorageModule).
coordinator | at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.metadata.storage.mysql.MySQLMetadataStorageModule)
coordinator |
coordinator | 2) A binding to org.apache.druid.server.audit.SQLAuditManagerConfig was already configured at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:152) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.metadata.storage.postgresql.PostgreSQLMetadataStorageModule).
coordinator | at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:152) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.metadata.storage.mysql.MySQLMetadataStorageModule)

Contributor guide

Open the contributing guide

Research direction

Start with SQLMetadataStorageDruidModule.configure and the MySQLMetadataStorageModule and PostgreSQLMetadataStorageModule mentioned in the stack trace. Reproduce the failure with both entries in druid.extensions.loadList, then inspect how the shared SQL audit configuration is bound when both extensions load. Done means the coordinator starts with both metadata extensions enabled and their SQL ingest modules available.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql, postgresql
Domain
backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.