awslabs / awslabs/amazon-dynamodb-tools

Employ shuffle scans to diffuse the read/write traffic of each task

Open
#90 0 comments 0 reactions 0 assignees View on GitHub
bulk_executor enhancement help wanted
Dominant language
Python
Stars
181
Forks
50
Avg merge
4h 11m
Merged PRs (30d)
36

Description

Today scan-oriented commands (`scancount`, `diff`) each are given a segment to use and do all their reading from that segment. It tends to direct read traffic to the same linear keyspace area and thus the same small set of partitions. Today we often rate limit a task to 1,500 read units per second to avoid creating overly hot partitions.

An improvement would be to use shuffle scans, where the task is handed N many segments and reads from them round-robin. That would spread the task's read traffic more fairly across more partitions.

This would also help with write commands (`update`) which today are driven by scans and would benefit from updating more partitions from the same task. Then we could raise the 500 write units per second as well.

Bottom line: pass to each task not a segment to use but a set of segments that are used within the task round-robin.

Today:
1,000 segments, 1,000 tasks each with 1 segment, 50 executors, so 50 segments being hit

Better:
1,000 segments, 200 tasks each with 5 randomly selected segments, 50 executors, so 250 segments being hit

Now, this makes each task take longer because each task is responsible for 5x as much table space. That can be solved by doing 5,000 segments instead, back to 1,000 tasks, with 5 random segments per task. Each task still does 0.1% of the table but divided into 5 disjointed chunks instead of 1 contiguous chunk.

In reality I might suggest sets of 10 segments per task, not actually 5.

Contributor guide

Open the contributing guide

Research direction

Start by locating the implementations of the scan-oriented commands scancount, diff, and update, then trace how each task receives its segment. Define the shuffle-scan behavior around tasks receiving disjoint sets of segments and reading them round-robin; done should include the intended distribution across partitions without changing the commands' purpose.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.