algorand / algorand/go-algorand

Add metrics collection to tracker caches

Open
#4,398 0 comments 0 reactions 0 assignees View on GitHub
Infrastructure Team Carbon-11 Team Scytale
Dominant language
Go
Stars
1.4k
Forks
537
Avg merge
1d 7h
Merged PRs (30d)
18

Description

## Problem

go-algorand dedicates a substantial amount of memory and code complexity to maintaining caches in the trackers registered with the trackerRegistry, so it's only natural that we should have some idea how well the caches are performing on test and production workloads.

Caches:
1. `onlineAccounts` tracker
- `baseOnlineAccounts`
- `onlineAccountsCache` (on superficial inspection, this cache implementation differs somewhat from the others)
2. `accountUpdates` tracker
- `baseAccounts`
- `baseResources`
- `baseKV` (available when box work is completed)

The following trackers don't maintain caches: `txTail`, `catchpointTracker`, `bulletin`, `blockNotifier`, and `metrics`.

Historically, the original purpose of most of these caches was to speed up writes to the corresponding database (see #1818). With that said, the caches also improve read performance and a regression here would impact important code paths so metrics describing both should be supported.

## Solution

Ideally, the exported metrics would cover at least these bases:
1. separate cache hit rate/miss rate metrics for reads, writes, and overall.
2. cache size both in terms of the number of elements and the total size of all elements in bytes.

PR #2115 can be used as a reference for how to add metrics.

## Dependencies

Metrics are typically consumed by a local Prometheus service. A jenkins job running pingpong tests can be used to confirm the metrics are being exported correctly.

## Urgency

Low-Medium: While caches can potentially have a very large impact on performance and resource consumption, they've been used in go-algorand for a while so metrics collection isn't needed in the immediate future. The urgency will most likely increase over time because, as the Algorand blockchain grows, the caches may reach their capacity leading to hidden issues surfacing.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.