Clarify query-backed Dataset.source representation and SQL dialect
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 267
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 24
Description
## Problem
The [core specification](https://github.com/apache/ossie/blob/main/core-spec/spec.md#datasets)
defines `Dataset.source` as a string that may identify a physical table/view or contain a query.
For a query-backed source, the specification provides neither:
- a discriminator that distinguishes a relation identifier from SQL text; nor
- a SQL dialect for parsing or validating the query.
Consumers therefore cannot interpret query-backed sources consistently.
## Questions
1. Is a query-backed `Dataset.source` intended to be portable core behavior?
2. If so, should the specification provide a structured form such as:
```yaml
source:
type: query
dialect: ANSI_SQL
query: SELECT ...
```
3. Should the existing string form remain the relation-identifier form for compatibility?
4. If query interpretation is intentionally consumer-specific, should the specification
explicitly allow consumers to reject query-backed sources?
## Context
Gravitino maps an OSI dataset source to a catalog Table or View. Because the query dialect is
currently unspecified, its proposed structured Metric View model requires a query to be registered
as a logical View before it can be referenced as a dataset source.
Related discussions: #52, #294, and #104.
Contributor guide
Research direction
Start with the Dataset section in core-spec/spec.md, then review related discussions #52, #294, and #104 for prior decisions. Determine whether the specification can define the source representation and SQL dialect, or must document consumer-specific handling; done means the open questions are resolved and the relevant specification text is updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100