redpanda-data / redpanda-data/connect

Memory cache never compacts expired keys with a stable set of cache keys

Open
#1,993 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

annoying caches ux
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

We're using Benthos to periodically monitor and process a set of IDs. To prevent needless reprocessing, we're caching IDs for some period of time. If the ID wasn't successfully processed downstream after the cache key TTL, we expect the ID to have been purged from the cache and be reprocessed.

Minimal configuration:

input:
  generate:
    interval: 1s
    mapping: root.key = "a"
pipeline:
  threads: -1
  processors:
    - cache:
        resource: mycache
        operator: add
        key: '${! json("key") }'
        value: "foo"
        ttl: 10s
    - mapping: root = if errored() { deleted() }
output:
  label: ""
  stdout:
    codec: lines
cache_resources:
  - label: mycache
    memory:
      compaction_interval: 1s

Expected output after 10s

{"key":"a"}
{"key":"a"}

Actual output after 10s

{"key":"a"}

The current work around is to always set a key on each pass through the pipeline to force a compaction for expired cache entries.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the minimal YAML configuration in the issue and observe the memory cache after the 10-second TTL with a stable key. Trace the memory cache compaction behavior and verify that expired keys are removed and the ID is emitted again without forcing a new key write.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.