Provide Redis metadata store that writes key/values instead of single hash [INT-4065]

Aaron Loes opened INT-4065 and commented

Took a look at the RedisMetadataStore for use with the IdempotentReceiverInterceptor for deduplication for our high throughput data streams. We noticed that it maintains a single hash object for the metadata. For this use case, having a single hash object for all metadata doesn't really work as we would blow out the memory of any server pretty quickly. Since we really don't care about data after a deterministic amount of time because we're really only trying to avoid reprocessing data we've already seen due to a replay scenario, ideally we would want the metadata to expire.

Being that hash keys cannot have an expiration on them (i believe) and because the current implementation doesn't allow for metadata expiration, a version that doesn't use a hash but rather individual key/value pairs for each metadata record would suffice as we could turn expiration from the server. But more ideal would be to set a TTL for records individually for differing needs.

I could see this in a few different forms.

1. an alternative implementation of the MetadataStore (RedisKeyValueMetadataStore) that uses key/value instead of hash and that also allows for TTL
   or
2. an implementation of the existing RedisMetadataStore that's takes in some sort of strategy for writing data to redis where one strategy would be to use a single hash and another would use key/value pairs prefixed with a key and accounted for TTL

Reference URL: https://stackoverflow.com/questions/45495329/using-redismetadatastore-redisproperties-with-a-redis-cluster

2 votes, 4 watchers

Start by reading the RedisMetadataStore and its use with IdempotentReceiverInterceptor to understand the existing single-hash behavior. The issue presents alternative designs, so a maintainer decision is needed before implementation; completion would require an agreed key/value metadata approach with per-record expiration.