Initial PySpark support
- Dominant language
- Java
- Stars
- 55
- Forks
- 18
- Avg merge
- 6h 29m
- Merged PRs (30d)
- 16
Description
- Implement read/write storage functions to handle both directory and single file formats
- Selection logic for deciding when to read inputs as Pandas vs PySpark
- Hooks for repartitioning / flattening between storage and presenting data to models
- Implement main context methods for get/put PySpark
- Implicit conversion - data items loaded as Pandas automatically converted if requested as PySpark, reverse is available with a row limit
- Run example PySpark model from doc folder as end-to-end validation
Contributor guide
Research direction
Start with the PySpark model example in the doc folder, then trace the existing storage functions and main context get/put methods it exercises. Validate directory and single-file reads and writes, Pandas/PySpark selection and conversions, repartitioning and flattening, and the documented example end to end.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100