apache / apache/accumulo

Exception during scan due to cached block eviction (Possible recurrence of issue described in #774)

Open
#1,973 9 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

**Bug Description**
It appears that we've encountered the same behavior as observed in #774 while testing our rel/2.0.1 deployment in AWS. For a period of time, we observed that our table scans would sporadically fail with the error below (maybe once or twice per hour over a period of a couple days of continuous query testing)
```
[scan.LookupTask] WARN : exception while doing multi-scan
java.lang.IllegalStateException: Block was evicted
at org.apache.accumulo.core.file.blockfile.cache.lru.CachedBlock.recordSize(CachedBlock.java:159)
at org.apache.accumulo.core.file.blockfile.cache.lru.LruBlockCache.cacheBlock(LruBlockCache.java:238)
at org.apache.accumulo.core.file.blockfile.cache.lru.LruBlockCache.cacheBlock(LruBlockCache.java:263)
at org.apache.accumulo.core.file.blockfile.cache.lru.SynchronousLoadingBlockCache.getBlock(SynchronousLoadingBlockCache.java:133)
at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:409)
at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.getIndexBlock(MultiLevelIndex.java:808)
at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.access$100(MultiLevelIndex.java:585)
at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.lookup(MultiLevelIndex.java:628)
at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader$Node.access$400(MultiLevelIndex.java:591)
at org.apache.accumulo.core.file.rfile.MultiLevelIndex$Reader.lookup(MultiLevelIndex.java:817)
at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader._seek(RFile.java:997)
at org.apache.accumulo.core.file.rfile.RFile$LocalityGroupReader.seek(RFile.java:895)
at org.apache.accumulo.core.iterators.system.LocalityGroupIterator.seek(LocalityGroupIterator.java:281)
at org.apache.accumulo.core.file.rfile.RFile$Reader.seek(RFile.java:1437)
at org.apache.accumulo.server.problems.ProblemReportingIterator.seek(ProblemReportingIterator.java:103)
at org.apache.accumulo.core.iterators.system.MultiIterator.seek(MultiIterator.java:104)
at org.apache.accumulo.core.iterators.system.StatsIterator.seek(StatsIterator.java:63)
at org.apache.accumulo.core.iterators.system.DeletingIterator.seek(DeletingIterator.java:73)
at org.apache.accumulo.core.iterators.ServerSkippingIterator.seek(ServerSkippingIterator.java:52)
at org.apache.accumulo.core.iterators.system.ColumnFamilySkippingIterator.seek(ColumnFamilySkippingIterator.java:127)
at org.apache.accumulo.core.iterators.SynchronizedServerFilter.seek(SynchronizedServerFilter.java:56)
at org.apache.accumulo.core.iterators.WrappingIterator.seek(WrappingIterator.java:99)
at org.apache.accumulo.core.iterators.user.VersioningIterator.seek(VersioningIterator.java:82)
at aquery.iterators.Query.get_pageid(Query.java:453)
at aquery.iterators.Query.next(Query.java:351)
at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.readNext(SourceSwitchingIterator.java:174)
at org.apache.accumulo.core.iterators.system.SourceSwitchingIterator.next(SourceSwitchingIterator.java:145)
at org.apache.accumulo.tserver.tablet.Tablet.lookup(Tablet.java:641)
at org.apache.accumulo.tserver.tablet.Tablet.lookup(Tablet.java:770)
at org.apache.accumulo.tserver.scan.LookupTask.run(LookupTask.java:116)
at org.apache.accumulo.tserver.session.ScanSession$ScanMeasurer.run(ScanSession.java:54)
at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:748)
```

**Versions:**
- **Accumulo:** 2.0.1
- **OS:** CentOS 7.7.1908
- **Java:** OpenJDK 1.8.0_272
- **Hadoop:** 3.0.0-cdh6.3.1
- **ZK:** 3.4.14

**Cluster Environment:**
- 12-node m5.2xlarge ec2 cluster; 10 tservers; 3 ZKs; Hadoop HA namenodes
- LRU cache config:
```
default | tserver.cache.data.size ........................... | 10%
site | @override ...................................... | 2G
system | @override ...................................... | 4G
default | tserver.cache.index.size .......................... | 25%
site | @override ...................................... | 2G
system | @override ...................................... | 1G
default | tserver.cache.manager.class ....................... | org.apache.accumulo.core.file.blockfile.cache.lru.LruBlockCacheManager
```

**Other relevant facts:**
- As best we can tell, the errors began to surface several hours after having switched `tserver.cache.manager.class` (via accumulo shell) from *[accumulo.ohc.OhcCacheManager](https://github.com/keith-turner/accumulo-ohc)* back to *LruBlockCacheManager*
- No intervening restart of Accumulo was performed after said config change in the shell
- At the time of the errors, the cluster was under medium-heavy load from concurrent user queries/scans and live ingest
- After a recent full restart of Accumulo, the errors seem to have ceased

There is no evidence in logs of this error having occurred prior to this instance (going back over a month), and several hours have lapsed since the full restart of Accumulo. No recurrence so far

Contributor guide

Open the contributing guide

Research direction

Start with the cached-block classes in the stack trace, especially CachedBlock.java, LruBlockCache.java, SynchronousLoadingBlockCache.java, and CachableBlockFile.java, and compare the behavior with issue #774. Reproduce the scan failure around switching to LruBlockCacheManager without a restart, using the listed cache settings and load conditions. Done means the cause is identified and scans no longer fail with Block was evicted.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, hadoop, java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.