asyml / asyml/forte

Misuse of the cache_key_function in several data readers

Open
#239 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
253
Forks
59
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
In several readers such as `ms_macro_passage_reader`, `conllu_ud_reader`, etc, the implementations of `cache_key_function` is wrong, as they all take `data_pack` as input argument. `cache_key_function` should take the output of `_collect` to do caching to help construct `data_pack` faster. If we already have a `data_pack`, then there is no need to query the cache.

**Expected behavior**
All the readers should perform the caching function properly.

**Additional context**
Works to consider:
1. Revisit the design of `cache_key_function`. Currently, it is a bit
confusing to implement and the documentation for this is not enough.
2. Design a general reader test that can be reused for all reader
classes, which will test the caching ability if caching is implemented.

Contributor guide

Open the contributing guide

Research direction

Review the cache_key_function implementations in readers such as ms_macro_passage_reader and conllu_ud_reader, along with the _collect flow. Clarify the cache input contract and inspect existing reader tests before designing a reusable caching test. Done means readers cache from _collect output, avoid redundant cache queries when a data_pack exists, and the behavior is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.