apache / apache/couchdb

Start big shards indexing with delay and load distribution

Open
#5,811 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Erlang
Stars
7k
Forks
1.1k
Avg merge
1d 16h
Merged PRs (30d)
9

Description

## Summary

I have a large shard which needs to be indexed using a new view. When the indexing operation started, the same shard was indexed on several nodes.

Image

This can be optimised using this logic.
1) If the shard has X changes, then the required delay is indexing for Y milliseconds. The delay may be larger on a node with a high CPU load and smaller on a node with less CPU load;
2) check the other cluster node started indexing for the same shard;
3) if the shard is already indexed on another node, then skip this shard for indexing now.

The same logic applied to all shards. At the end, all nodes are randomly distributed shards for indexing.
On the next view request, the same shard will be indexed on the other node.

## Desired Behaviour

Nodes distribute the shards indexing task and do not index the same shard on several nodes at the same time.

## Additional context

This allows for distributing CPU load on nodes and makes CPU spikes smoother.

Contributor guide

Open the contributing guide

Research direction

Start by locating the shard indexing workflow and the cluster-node coordination logic described in the issue. Review how indexing changes are counted and how a node determines whether another node is indexing the same shard. Done means indexing work is delayed and distributed so nodes do not index the same shard concurrently, with CPU load considered.

Written by the indexing model from the issue text.

Assessment

Tech stack
erlang
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.