Investigate high-performance data serialization
- Dominant language
- Python
- Stars
- 253
- Forks
- 59
- PR merge metrics
- No merged PRs in 30d
Description
This issue should be done at the end of project https://github.com/asyml/forte/projects/3.
**Is your feature request related to a problem? Please describe.**
Forte currently supports a simple serialization via JsonPickle. This format is fairly readable but the performance for loading and serialization are not optimized. Furthermore, we don't have control over the whole serialization process, which stops us from performing checking during the serialization
This issue needs to be further clarified after the main data structure is defined.
**Describe the solution you'd like**
1. Investigate the alternatives and create a rough estimation
2. Implement a different serialization that uses the new method.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
- This is part of the data efficiency project
- This PR should be made to the `new_datapack` branch.
Contributor guide
Research direction
Start by reviewing the data efficiency project and the main data structure definition, then examine the current JsonPickle serialization path. Compare serialization alternatives and record a rough effort estimate before implementation. Done means a replacement serialization method is implemented on the new_datapack branch with control over the process and checking during serialization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100