asyml / asyml/forte

Integrate `DataStore` with `Payload` entry

Open
#847 1 comment 0 reactions 0 assignees View on GitHub
data_efficiency
Dominant language
Python
Stars
253
Forks
59
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
This is a follow-up issue to #840. After we restructure the payload system in #828, we have a new ontology called `Payload` and it's not entirely integrated to `DataStore` yet. Currently `DataPack` stores three payload lists for different modalities. We want to migrate them into `DataStore` without affecting the current interfaces.

**Depends on**
- #840

**Describe the solution you'd like**
* We should maintain all payload lists in `DataStore`
* `__getitem__` should be efficient
* Design how `Payload`'s attributes can be stored in `DataStore` as primitive and built-in types.
* Need to determine how to handle the `dataclass` and non-`dataclass` fields.
* Remove `__getstate__` and `__setstate__` from `Payload`
* We should keep the previous interfaces of accessing payloads, e.g., `pack.text_payloads[text_payload_index]`

**Additional context**
- This is part of the data efficiency project
- This PR should be made to the `master` branch.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the follow-up work in #840 and the payload restructuring in #828, then inspect the existing DataPack, Payload, and DataStore entry points. Determine how all payload lists and dataclass or non-dataclass attributes can be represented while preserving accesses such as pack.text_payloads[index]. Done means the interfaces remain compatible, DataStore handles efficient __getitem__, and Payload no longer needs __getstate__ or __setstate__.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.