stac-utils / stac-utils/rustac-py
Better UX for non-schema-matching geoparquet batches
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54
- Forks
- 5
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
From an internal chat:
i'm curious how we can make a good UX around non-schema-matching batches. if you're doing a large workflow, it'd be really nice to "peel off" all the non-compliant items, put them into that initial schema-inference batch, and then re-run
This should be doable:
- Down in Rust, the writer can accumulate all failed batches ... this could lead to a big memory blow-up, so maybe instead we should just return the failed batch right away?
- The user could them use add failed batch to the initial "schema inference" batch and re-run
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Rust writer's handling of failed batches and the Python path for the initial schema-inference batch. Clarify whether a failed batch should be returned immediately or accumulated, then verify that non-compliant items can be added to the inference batch and rerun without unbounded memory use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100