Integrate `DataStore` with `Payload` entry
- Dominant language
- Python
- Stars
- 253
- Forks
- 59
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
This is a follow-up issue to #840. After we restructure the payload system in #828, we have a new ontology called `Payload` and it's not entirely integrated to `DataStore` yet. Currently `DataPack` stores three payload lists for different modalities. We want to migrate them into `DataStore` without affecting the current interfaces.
**Depends on**
- #840
**Describe the solution you'd like**
* We should maintain all payload lists in `DataStore`
* `__getitem__` should be efficient
* Design how `Payload`'s attributes can be stored in `DataStore` as primitive and built-in types.
* Need to determine how to handle the `dataclass` and non-`dataclass` fields.
* Remove `__getstate__` and `__setstate__` from `Payload`
* We should keep the previous interfaces of accessing payloads, e.g., `pack.text_payloads[text_payload_index]`
**Additional context**
- This is part of the data efficiency project
- This PR should be made to the `master` branch.
Contributor guide
Research direction
Start by reviewing the follow-up work in #840 and the payload restructuring in #828, then inspect the existing DataPack, Payload, and DataStore entry points. Determine how all payload lists and dataclass or non-dataclass attributes can be represented while preserving accesses such as pack.text_payloads[index]. Done means the interfaces remain compatible, DataStore handles efficient __getitem__, and Payload no longer needs __getstate__ or __setstate__.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100