finos / finos/tracdap

Initial PySpark support

Open
#28 0 comments 0 reactions 0 assignees View on GitHub
enhancement runtime-python
Dominant language
Java
Stars
55
Forks
18
Avg merge
6h 29m
Merged PRs (30d)
16

Description

- Implement read/write storage functions to handle both directory and single file formats
- Selection logic for deciding when to read inputs as Pandas vs PySpark
- Hooks for repartitioning / flattening between storage and presenting data to models
- Implement main context methods for get/put PySpark
- Implicit conversion - data items loaded as Pandas automatically converted if requested as PySpark, reverse is available with a row limit
- Run example PySpark model from doc folder as end-to-end validation

Contributor guide

Open the contributing guide

Research direction

Start with the PySpark model example in the doc folder, then trace the existing storage functions and main context get/put methods it exercises. Validate directory and single-file reads and writes, Pandas/PySpark selection and conversions, repartitioning and flattening, and the documented example end to end.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.