OpenML API parquet migration - Phase 2
Open
Nobody has claimed this yet.
CoreSystem
priority: high
- Dominant language
- PHP
- Stars
- 755
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
We already have dataset download support in parquet and MinIO, now the next phase is uploading these datasets.
We need to allow parquet upload directly to MinIO. For this there are 3 components which are needed to be changed:
- OpenML client APIs(python/R/Java): To convert dataset directly from dataframe to parquet and send an upload request.
- OpenML API: Assign the uploaded dataset ID and then transfer it to the MinIO. (we already have scripts for transfer)
- OpenML Evaluation engine, to process the parquet datasets.
@PGijsbers @joaquinvanschoren @janvanrijn
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the three areas named in the issue: the Python, R, and Java client APIs, the OpenML API and its existing transfer scripts, and the evaluation engine. Confirm the upload flow converts dataframes to parquet, assigns a dataset ID and transfers data to MinIO, and that the evaluation engine processes the resulting parquet datasets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python, r
- Domain
- api, data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100