Misuse of the cache_key_function in several data readers
- Dominant language
- Python
- Stars
- 253
- Forks
- 59
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
In several readers such as `ms_macro_passage_reader`, `conllu_ud_reader`, etc, the implementations of `cache_key_function` is wrong, as they all take `data_pack` as input argument. `cache_key_function` should take the output of `_collect` to do caching to help construct `data_pack` faster. If we already have a `data_pack`, then there is no need to query the cache.
**Expected behavior**
All the readers should perform the caching function properly.
**Additional context**
Works to consider:
1. Revisit the design of `cache_key_function`. Currently, it is a bit
confusing to implement and the documentation for this is not enough.
2. Design a general reader test that can be reused for all reader
classes, which will test the caching ability if caching is implemented.
Contributor guide
Research direction
Review the cache_key_function implementations in readers such as ms_macro_passage_reader and conllu_ud_reader, along with the _collect flow. Clarify the cache input contract and inspect existing reader tests before designing a reusable caching test. Done means readers cache from _collect output, avoid redundant cache queries when a data_pack exists, and the behavior is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100