kestra-io / kestra-io/plugin-databricks

DatabricksSQLCLI declares NamespaceFilesInterface but namespaceFiles has no effect

Open
#260 0 comments 0 reactions 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
3
Forks
10
Avg merge
2d 59m
Merged PRs (30d)
9

Description

# DatabricksSQLCLI declares NamespaceFilesInterface but `namespaceFiles` has no effect

## Summary

`io.kestra.plugin.databricks.cli.DatabricksSQLCLI` declares `implements … NamespaceFilesInterface` ([`DatabricksSQLCLI.java:61`](https://github.com/kestra-io/plugin-databricks/blob/main/src/main/java/io/kestra/plugin/databricks/cli/DatabricksSQLCLI.java#L61)), so the property appears supported, but there is no `namespaceFiles` field on the class, `getNamespaceFiles()` is hardcoded to `return null` (line 205), and `run()` never calls `CommandsWrapper.withNamespaceFiles(...)`. A user who sets `namespaceFiles:` on the task gets no namespace files staged into the working directory.

## Actual Behaviour

Setting `namespaceFiles` on `DatabricksSQLCLI` does not stage any namespace file into the CLI working directory. Because no backing field exists, the property is either rejected at flow validation as an unknown property or silently dropped at deserialization — either way the declared capability does nothing, and the failure surfaces only as a "file not found" from `dbsqlcli` (or as a query that silently reads nothing).

```java
// DatabricksSQLCLI.java:204-207
@Override
public NamespaceFiles getNamespaceFiles() {
return null;
}
```

`run()` (lines 132-152) wires `.withInputFiles(inputFiles)` and `.withOutputFiles(...)` into `CommandsWrapper`, but there is no corresponding `.withNamespaceFiles(...)` call.

**Note — `inputFiles` is NOT affected.** `getInputFiles()` also returns `null` (line 200), but the `inputFiles` field does exist (line 127) and `run()` passes it directly to `CommandsWrapper` at line 149, so `inputFiles` works as documented. Please don't spend time chasing it. The `getInputFiles()` override is still a latent hazard — any core code path that reads through `InputFilesInterface` rather than the field will see nothing — so it is worth fixing in the same pass, but it is not a user-visible break today.

## Expected Behaviour

Either of the following, whichever matches the intent for this task — this is a product call, not just a code fix:

1. **Support it** — add a `private NamespaceFiles namespaceFiles;` field with `@PluginProperty(group = "source")` and a `@Schema`, return it from `getNamespaceFiles()`, and pass it through with `.withNamespaceFiles(runContext, namespaceFiles)` in `run()`. Same for making `getInputFiles()` return the actual field.
2. **Stop advertising it** — drop `NamespaceFilesInterface` from the `implements` clause so the schema no longer offers a property the task ignores.

Option 1 is preferable if namespace-file staging is useful here (it lets users keep `.sql` files in a namespace and run them via the CLI), which seems likely for this task.

## Reproducer

```yaml
id: databricks_sqlcli_namespacefiles
namespace: company.team

tasks:
- id: run_from_namespace_file
type: io.kestra.plugin.databricks.cli.DatabricksSQLCLI
host: "{{ secret('DATABRICKS_HOST') }}"
token: "{{ secret('DATABRICKS_TOKEN') }}"
httpPath: "{{ secret('DATABRICKS_HTTP_PATH') }}"
namespaceFiles:
enabled: true
commands:
- "SELECT 1"
```

Expected: namespace files are staged into the working directory before `dbsqlcli` runs.
Actual: nothing is staged; the property has no effect.

## Logs / Stack Trace

No logs available — the property is silently ineffective rather than failing loudly, which is part of the problem.

## Environment

- **Kestra version**:
- **Plugin version**: reproduced against `main` (as of PR #259 review)
- **Deployment**: any

## Additional Context

Found during the documentation review in https://github.com/kestra-io/plugin-databricks/pull/259, which added `@Schema` annotations to `inputFiles`/`outputFiles` and surfaced this gap. That PR is documentation-only and deliberately does not change runtime behaviour.

Acceptance criteria for the fix:

- [ ] `namespaceFiles` either works end-to-end or is no longer advertised via the interface
- [ ] If supported: property carries `@Schema` and `@PluginProperty(group = "source")`
- [ ] `getInputFiles()` returns the actual `inputFiles` field rather than `null`
- [ ] Unconditional happy-path test covering namespace-file staging (WireMock or Testcontainers — not an `@EnabledIfEnvironmentVariable`-gated test, which would be silently skipped in CI)

---
*[View as Artifact](https://claude.ai/code/artifact/d3d0277e-6ee7-46a5-8811-5d13e102c011)*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/main/java/io/kestra/plugin/databricks/cli/DatabricksSQLCLI.java, especially the namespaceFiles and inputFiles accessors and run() wiring, then inspect CommandsWrapper.withNamespaceFiles(...). Decide whether to support namespace-file staging or remove the advertised interface. Done means the chosen behavior is reflected in the schema and an unconditional happy-path test covers staging, using WireMock or Testcontainers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.