pathwaycom / pathwaycom/pathway
Nested `pw.Schema`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 62.3k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
Is your feature request related to a problem? Please describe.
Currently, there is no way to define a nested schema using pw.Schema.
Describe the solution you'd like
Be able to use nested pw.schema:
class NestedColumnSchema(pw.Schema):
a: int
b: float
class InputSchema(pw.Schema):
key: int
nested_column: NestedColumnSchema
Describe alternatives you've considered
Currently, a workaround is to use pw.JSON, but the engine cannot verify the schema inside it.
Additional context
Link to issue https://github.com/pathwaycom/pathway/issues/117
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the pw.Schema and pw.JSON implementations, then read the related issue #117 for context. Trace how schema fields are represented and validated, and confirm the example with NestedColumnSchema works while validating the nested fields instead of treating them as unrestricted JSON.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100