apache / apache/gravitino

[Improvement] Add parameter metadata to JobTemplate to support required/optional declaration

Open
#12,483 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

### What would you like to be improved?

When running a job via the Gravitino Web UI (CreateJobDialog), all template parameters extracted from {{placeholder}} tokens are treated as required — there is no way to mark a parameter as optional. This forces users to fill in values that are semantically unnecessary, and can cause incorrect job execution if a non-applicable value is provided.

Concrete example: builtin-iceberg-rewrite-data-files
```
The built-in IcebergRewriteDataFilesJob template declares these arguments:
// IcebergRewriteDataFilesJob.java — buildArguments()
"--catalog", "{{catalog_name}}"
"--table", "{{table_identifier}}"
"--strategy", "{{strategy}}"
"--sort-order", "{{sort_order}}"
"--where", "{{where_clause}}"
"--options", "{{options}}"
"--spark-conf", "{{spark_conf}}"
```

Among these, only catalog_name and table_identifier are truly required. The rest are optional:

Parameter | Required? | Notes
-- | -- | --
catalog_name | ✅ | Iceberg catalog name
table_identifier | ✅ | Target table (db.table)
strategy | ❌ | Defaults to binpack when omitted
sort_order | ❌ | Only meaningful when strategy => 'sort'; ignored by Iceberg when strategy => 'binpack'
where_clause | ❌ | Filter predicate
options | ❌ | JSON map of rewrite options
spark_conf | ❌ | Custom Spark configs

The Java backend correctly handles empty values. In IcebergRewriteDataFilesJob.buildProcedureCall():

```
if (sortOrder != null && !sortOrder.isEmpty()) {
sql.append(", sort_order => '")...
}
```

And Gravitino's own GravitinoCompactionJobAdapter explicitly passes an empty string for sort_order when using binpack:

```
return ImmutableMap.of(
"sort_order", "", // empty — binpack doesn't need it
"strategy", "binpack",
...
);
```

But the frontend forces all to be required
In web-v2/web/src/app/jobs/CreateJobDialog.js, getPlaceholderEntries() extracts all {{xxx}} tokens from the template, and the form applies a blanket required: true rule to every parameter's value field:

```

```

Image

There is no mechanism to distinguish required from optional parameters. Users are forced to fill in sort_order even when using strategy=binpack, where it has no effect.

The JobTemplate API defines arguments as a plain List:

```
// JobTemplate.java
protected final List arguments;

public List arguments() { return arguments; }
```

There is **no metadata** to indicate:

1. Which placeholders are required vs optional
2. Descriptions or default values for placeholders
3. Validation rules (e.g., "sort_order is required when strategy=sort")
This information is only documented in Javadoc, not encoded in the template structure.

### How should we improve?

Introduce a structured parameter descriptor so template authors can declare which placeholders are optional, along with descriptions and default values.

e.g.
```
public class TemplateParameter {

private final String name;
private final boolean required;
private final String description;
private final String defaultValue;
}
```
Frontend impact
CreateJobDialog.js would read parameters from the template response and apply required validation per-parameter instead of the blanket required: true:

```
// Instead of blanket required for all:
rules={[{ required: param.required, message: 'Please enter the job config value!' }]}
```

I'm willing to work on this. Would love to hear maintainers' thoughts on the API shape

Contributor guide

Open the contributing guide

Research direction

Start by reading JobTemplate.java and getPlaceholderEntries() and the form rules in CreateJobDialog.js, then compare IcebergRewriteDataFilesJob.buildArguments() with buildProcedureCall(). Done means an agreed structured parameter representation is exposed in the template response and the dialog uses per-parameter required metadata, descriptions, and defaults without forcing optional values.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, javascript
Domain
backend-api-design, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.