asyml / asyml/forte

Allow storing actual data content and large meta-data as additional references

Open
#416 0 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted priority: medium topic: data
Dominant language
Python
Stars
253
Forks
59
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Currently, the whole data pack is stored in one place, which includes the metadata, the actual data, and optionally the creation records. Serializing and deserializing such data is a very time-consuming process. In cases like the initialization stage, we only need to check the meta-data for basic validations, thus loading the actual data content would be a waste of time.

**Describe the solution you'd like**
A feature that stores the real data content, or additional creation records as separate files/pointers on a disk would be great, then doing loading, the system can "lazy load" just the meta data first, and then only take the actual data when requested.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. First trace how data packs are serialized, deserialized, and validated, then define how metadata, actual data, and creation records should be stored and loaded separately; done means metadata can be loaded without eagerly loading the other content.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.