Streamline the implementation of collect and caching in readers
- Dominant language
- Python
- Stars
- 253
- Forks
- 59
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
The design of `collect` and `cache_key_function` is to help develop a caching function during data reading. This design expects the users to cache the results of `collect`. and use the `cache_key` to retrieve it. But this creates the following problem:
1. If one does not want to develop caching, this design is confusing and cumbersome to them. Even worse, they will arbitrarily implement these functions, which are not effective for caching, providing bad examples for future adopters.
2. The test coverage for these is not enough.
**Describe the solution you'd like**
1. We probably need to provide two ways to implement readers: a simple reader, or a reader with some caching ability.
1. as requested several times, some base class for reader tests should be implemented, that automatically test these function branches (e.g. caching).
1. And we should provide standard implementations for readers.
**Describe alternatives you've considered**
**Additional context**
Some issues are related to this, such as this one: https://github.com/asyml/forte/issues/239
Contributor guide
Research direction
Start by reviewing the reader API and existing implementations of collect and cache_key_function, then read the related discussion in issue 239. Compare the current caching branches and test coverage before deciding how simple and caching-capable readers should differ. Done means the reader approaches and standard implementations are defined, with base tests covering the caching behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100