apache / apache/incubator-xtable

Generate a JSON Schema for datasetConfig from DatasetConfig

Open
#890 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Feature Request / Improvement

Publish a JSON Schema for `datasetConfig.yaml`, generated from the `DatasetConfig` class rather than written by hand.

## Why

Writing a valid `datasetConfig` is the first thing a new user does and the first place they get stuck: `sourceFormat`, `targetFormats`, `tableBasePath`, `tableName`, `namespace`, `partitionSpec`, and the right `conversionSourceProviderClass`. Today the only description of that shape is prose in the docs, so it drifts from the code silently.

A generated schema gives three things at once:

- Editors and IDEs validate and complete the file.
- Any agent or script can check a config before running a sync, instead of discovering problems from a stack trace.
- The documentation stops being a second source of truth that can go stale.

PR #829 includes a hand-written `config-schema.md` for exactly this purpose. That is the thing to replace with something generated.

## Scope

- Generate a JSON Schema from `DatasetConfig` and its nested types at build time.
- Publish it as a release artifact and at a stable URL, so `$schema` references keep working.
- Fail the build when the schema is out of date with the model, so it cannot silently drift.
- Reference it from the docs in place of any hand-maintained field list.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the DatasetConfig class and its nested types, then inspect the hand-written config-schema.md from PR #829 and the datasetConfig.yaml documentation. Define how the build generates and validates the schema, publishes it as a release artifact at a stable URL, and replaces the maintained field list in the docs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, documentation, release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.