apache / apache/ignite

Near Cache ignores entries read from back-up partitions

Open
#11,907 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
5.1k
Forks
1.9k
Avg merge
3d 2h
Merged PRs (30d)
46

Description

We have a simple 3 nodes cluster and we wanted to move read heavy data to ignite and use near cache to optimize cache read operations. After PoC with near caches we noticed that requested item not always saved in near cache even though near cache limit is not yet reached. I have run a simple scenario:
Test scenario:
1. Simple 3 nodes ignite cluster on bare metal hosts with mostly default configs:
- Ignite version 2.16 & 2.17
- Static discovery SPI of three IPs & ports
- Static communication SPI
- Discovery and Cache events are enabled
- Custom storage path for persistent regions
- two regions defined: default one is in memory:
- in memory with initial and max size set to 356/1024 MB and RANDOM_2_LRU eviction
- persistent one with initial and max size set to 64/512MB
3. Having near cache created with limit of 100 items and FIFO expiry
4. Request 100 unique items from cache (items saved in cache from different client)
5. Check near cache state with peaking it explicitly and simple get operation

Expected behavior:
- Every time item requested by key and absent from local near cache, it gets added to it. So that next request should always read it from local instance (considering it is not evicted in the meantime)
- Every update for locally stored item also reflected in local near cache

I found the following behavior:
- Having partitioned cache with zero back-ups - everything works as expected
- Having partitioned cache with two copies - only around 30% of requested items saved in cache. It is a complete random when item might be added to local cache, it can be after 1 or after 10 requests
- when after couple of rounds of request of the same set of keys cache finally gets full, switch to another set of keys causes eviction of items but new keys again not always added, so local cache shrinks
- Having replicated cache has same results on the same 3 server nodes set-up
- Having partitioned cache with back-ups with readFromBackup flag set to false - works as expected
- Having replicated cache with copies with readFromBackup flag set to false - properly saves to near cache but never reads from it when using plain get method. But it works fine when using explicit peak into near cache
- In each scenario near cache properly saves items that are created from the given client
- All scenarios work fine when all 3 servers and 1 client nodes are collocates on single host
- All scenarios have the same results for in-memory or persistent caches

So my assumption is that near cache for whatever reason not saving entries read from back-up partitions.
I have reviewed ignite documentation and haven't found any specific config required for Near Cache to work. I wonder if you can help me to identify what is wrong.

Sample cluster server config:
```java
IgniteConfiguration config = new IgniteConfiguration();

TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder()
.setAddresses(List.of(":22222", "host2.ip:22222", "host3.ip:22222"));

config.setDiscoverySpi(new TcpDiscoverySpi()
.setIpFinder(ipFinder)
.setLocalPort(22222)
.setLocalAddress(ipAddress)
.setLocalPortRange(0)
);

config.setCommunicationSpi(new TcpCommunicationSpi()
.setLocalPortRange(0)
.setLocalAddress(ipAddress)
);

config.getDataStorageConfiguration()
.setDefaultDataRegionConfiguration(
new DataRegionConfiguration()
.setInitialSize(356 * 1024 * 1024)
.setMaxSize(1024 * 1024 * 1024)
.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU)
).setDataRegionConfigurations(
new DataRegionConfiguration()
.setName("persistence")
.setInitialSize(64* 1024 * 1024)
.setMaxSize(512* 1024 * 1024)
.setPersistenceEnabled(true)
);
```
Sample cache config
```java
NearCacheConfiguration nearCacheConfiguration = new NearCacheConfiguration<>();
nearCacheConfiguration.setNearEvictionPolicyFactory(new FifoEvictionPolicyFactory<>(100));

CacheConfiguration cacheConfiguration = new CacheConfiguration<>(cacheName);
cacheConfiguration
.setCacheMode(CacheMode.PARTITIONED)
.setBackups(2)
.setDataRegionName("persistence")
.setNearConfiguration(nearCacheConfiguration);
```
Depends if cache is already present or a not we use `ignite.createCache` or `ignite.createNearCache`

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the three-node scenarios using the shown CacheConfiguration and NearCacheConfiguration, comparing backups, replicated caches, and readFromBackup settings. Inspect the near-cache behavior reached through ignite.createCache and ignite.createNearCache, and verify whether repeated gets populate and then hit the local near cache as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.