asyml / asyml/forte

Streamline the implementation of collect and caching in readers

Open
#262 0 comments 0 reactions 0 assignees View on GitHub
enhancement priority: medium topic: data topic: interface
Dominant language
Python
Stars
253
Forks
59
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
The design of `collect` and `cache_key_function` is to help develop a caching function during data reading. This design expects the users to cache the results of `collect`. and use the `cache_key` to retrieve it. But this creates the following problem:

1. If one does not want to develop caching, this design is confusing and cumbersome to them. Even worse, they will arbitrarily implement these functions, which are not effective for caching, providing bad examples for future adopters.
2. The test coverage for these is not enough.

**Describe the solution you'd like**
1. We probably need to provide two ways to implement readers: a simple reader, or a reader with some caching ability.
1. as requested several times, some base class for reader tests should be implemented, that automatically test these function branches (e.g. caching).
1. And we should provide standard implementations for readers.

**Describe alternatives you've considered**

**Additional context**
Some issues are related to this, such as this one: https://github.com/asyml/forte/issues/239

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the reader API and existing implementations of collect and cache_key_function, then read the related discussion in issue 239. Compare the current caching branches and test coverage before deciding how simple and caching-capable readers should differ. Done means the reader approaches and standard implementations are defined, with base tests covering the caching behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.