redpanda-data / redpanda-data/connect
Memory cache never compacts expired keys with a stable set of cache keys
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
We're using Benthos to periodically monitor and process a set of IDs. To prevent needless reprocessing, we're caching IDs for some period of time. If the ID wasn't successfully processed downstream after the cache key TTL, we expect the ID to have been purged from the cache and be reprocessed.
Minimal configuration:
input:
generate:
interval: 1s
mapping: root.key = "a"
pipeline:
threads: -1
processors:
- cache:
resource: mycache
operator: add
key: '${! json("key") }'
value: "foo"
ttl: 10s
- mapping: root = if errored() { deleted() }
output:
label: ""
stdout:
codec: lines
cache_resources:
- label: mycache
memory:
compaction_interval: 1s
Expected output after 10s
{"key":"a"}
{"key":"a"}
Actual output after 10s
{"key":"a"}
The current work around is to always set a key on each pass through the pipeline to force a compaction for expired cache entries.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the minimal YAML configuration in the issue and observe the memory cache after the 10-second TTL with a stable key. Trace the memory cache compaction behavior and verify that expired keys are removed and the ID is emitted again without forcing a new key write.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100