HewlettPackard / HewlettPackard/foedus_code

Compact in-memory epoch-history in logger

Open
#19 0 comments 0 reactions 1 assignee Claimed by @hkimura View on GitHub
enhancement
Dominant language
C++
Stars
64
Forks
27
PR merge metrics
No merged PRs in 30d

Description

Currently we keep every single history even though they are thousands of epochs earlier.
We don't need to keep all of them in-memory. We can discard 9/10 of them, for instance.
If a gleaner is lagging behind, it might have to scan more logs due to the history compaction, so we have to find a right balance between wasted memory and wasted work.

Or, write the history out to a file.

related code:

void LoggerRef::add_epoch_history(const EpochMarkerLogType& epoch_marker) {
...
if (control_block_->epoch_history_count_ >= LoggerControlBlock::kMaxEpochHistory) {
// TASK(Hideaki) To avoid this, we should maintain sparse history, like one history per
// tens of epochs. So far kMaxEpochHistory is enough big, so it won't happen.
LOG(FATAL) << "Exceeded kMaxEpochHistory. Unexpected.";
}

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.