apache / apache/pinot

Cached data accessor may be inconsistent even in the same controller

Open
#4,665 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

In https://github.com/apache/incubator-pinot/pull/3633, we leveraged `ZkCacheBaseDataAccessor` to cache instance configs in order to reduce the zk access.
While it has its drawback that it's only used to read instance config; writing/updating instance config is done in Helix code and it isn't controlled by Pinot. Thus, it's possible that the cached instance configs may be stale.
Thus, the current workaround is only use cached accessor for segment operations. For tenant/table level operations, we use direct data accessor. It's because segment operations are more frequent than the ones of tenant/table.
In the future, the following actions can be done for this issue:
* Contact Helix team to make `updateCache` method public in cached data accessor.
* Maintain a zkClient in PinotHelixResourceManager and override the methods like addInstance.
* Implement our own guava cache to cache instance configs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.