apache / apache/arrow-java

[Java][FlightSQL] JDBC Driver fails when Looker formats the URL with ':direct=' instead of '://'

Open
#1,203 7 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Java
Stars
94
Forks
152
Avg merge
3d 16h
Merged PRs (30d)
11

Description

### Describe the enhancement requested

I am running a customer-hosted Looker instance and trying to connect to Dremio using the
Apache Arrow Flight SQL JDBC driver (`flight-sql-jdbc-driver`), registered in Looker via a
`custom_jdbc_config.yml` entry so the native Dremio dialect can be kept:

```yaml
- name: dremio
dir_name: dremio
module_path: org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver
override_jdbc_url_subprotocol: arrow-flight-sql
```

`override_jdbc_url_subprotocol` only swaps the *subprotocol token* Looker plugs into its
existing, dialect-specific URL template for Dremio — it does not rewrite the rest of the
template. Looker's native Dremio dialect builds connection strings as:

```
jdbc:dremio:direct=:
```

so after the subprotocol override, Looker generates:

```
jdbc:arrow-flight-sql:direct=:
```

instead of the driver's expected format:

```
jdbc:arrow-flight-sql://:
```

Because `ArrowFlightJdbcDriver` parses the URL with `java.net.URI.create(...)` and strictly
expects the `://` delimiter to separate scheme from host, it cannot extract a host from the
`:direct=` form, and the connection fails.

I tried a workaround by switching Looker's dialect dropdown to "Trino" (which emits the
standard `://` format). While queries did run correctly over Arrow Flight, it broke the
Looker SQL Runner UI, since Looker then sends Trino-specific metadata queries to Dremio.

To use this driver properly with Looker, the native Dremio dialect needs to stay selected,
which means the driver needs to tolerate Looker's `:direct=` URL format.

### Steps to Reproduce / Current Behavior

Passing a connection string containing `:direct=` to the driver:

```
jdbc:arrow-flight-sql:direct=:
```

fails instantly with:

```text
Java::JavaSql::SQLException: URL must have a host. Expected format: jdbc:arrow-flight-sql://[host][:port][?param1=value&...]
```

### Fix

Implemented and verified locally against `flight/flight-sql-jdbc-core`. Added a normalization
step to `ArrowFlightJdbcDriver.java` that rewrites the legacy `:direct=` delimiter to `://`
before any URL parsing occurs, applied consistently at every entry point that parses the
connection string (`acceptsURL`, `connect`, `getUrlsArgs`):

```java
// Some BI tools (e.g. Looker) generate connection strings using the legacy
// "jdbc:arrow-flight-sql:direct=:" syntax instead of the standard "://" delimiter.
// Normalize it so the URL can still be parsed as a valid URI.
private static final String LEGACY_DIRECT_DELIMITER = ":direct=";

private static String normalizeUrl(final String url) {
if (url != null && url.contains(LEGACY_DIRECT_DELIMITER)) {
return url.replace(LEGACY_DIRECT_DELIMITER, "://");
}
return url;
}
```

- `acceptsURL(String url)` now normalizes before checking the `jdbc:arrow-flight-sql://` /
`jdbc:arrow-flight://` prefixes.
- `connect(String url, Properties info)` normalizes once up front and uses the normalized URL
for both argument parsing (`getUrlsArgs`) and `ArrowFlightConnection.createNewConnection`.
- `getUrlsArgs(String url)` normalizes defensively as well, since it is `@VisibleForTesting`
and may be called directly.

This is a purely additive, backward-compatible change — URLs that don't contain `:direct=`
are returned unmodified.

### Testing / Verification

Added two regression tests to `ArrowFlightJdbcDriverTest`:

- `testDriverUrlParsingMechanismShouldAcceptLegacyDirectDelimiter` — verifies `acceptsURL` and
`getUrlsArgs` correctly parse host/port/params from a `jdbc:arrow-flight-sql:direct=...` URL.
- `testShouldConnectWhenProvidedWithLegacyDirectDelimiterUrl` — verifies a live connection can
be established end-to-end using the `:direct=` URL format against the test Flight SQL server.

Full `ArrowFlightJdbcDriverTest` suite: 18/18 tests passing, 0 failures, 0 errors.

Also validated end-to-end against a real customer-hosted Looker instance connecting to Dremio
with the native Dremio dialect (config above) — connection succeeded after applying the fix.

### Additional deployment note (not part of this fix, but relevant to anyone testing it with Looker)

The `flight-sql-jdbc-driver` shaded jar is built with Java 17 bytecode by default
(`maven.compiler.release=17` in the root POM), but Looker's bundled JRE is Java 11. Loading
the default build in Looker fails with:

```
UnsupportedClassVersionError: ... class file version 61.0, this version of the Java Runtime
only recognizes class file versions up to 55.0
```

To build a Looker-compatible jar, rebuild the driver and its Arrow-owned dependency closure
(`arrow-format`, `arrow-memory-core`, `arrow-memory-netty(-buffer-patch)`, `arrow-vector`,
`flight-core`, `flight-sql`, `flight-sql-jdbc-core`, `flight-sql-jdbc-driver`) with:

```bash
mvn -pl format,memory/memory-core,memory/memory-netty,memory/memory-netty-buffer-patch,vector,flight/flight-core,flight/flight-sql,flight/flight-sql-jdbc-core,flight/flight-sql-jdbc-driver \
-DskipTests -Dmaven.compiler.release=11 -Dmaven.compiler.source=11 -Dmaven.compiler.target=11 \
clean install
```

(`clean` is required — Maven's incremental compiler otherwise skips recompilation when only
compiler flags change.) This may be worth calling out in the driver's documentation for users
deploying it to JVM-hosted BI tools that ship an older bundled runtime.

Contributor guide

Open the contributing guide

Research direction

Start in flight/flight-sql/flight-sql-jdbc-driver at ArrowFlightJdbcDriver.java, then read ArrowFlightJdbcDriverTest and its URL-parsing tests. Reproduce the legacy :direct= case and verify that acceptsURL, getUrlsArgs, and connect handle the host, port, and parameters while the existing suite remains passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.