awslabs / awslabs/amazon-dynamodb-tools

Have `update` do a scan phase then update phase to spread writes

Open
#87 1 comment 0 reactions 0 assignees View on GitHub
bulk_executor enhancement help wanted
Dominant language
Python
Stars
181
Forks
50
Avg merge
4h 11m
Merged PRs (30d)
36

Description

Working with a bulk update, we noticed on a table with 7 million items there were 5 million with the same PK value. The segment having that PK value became the long pole in the execution time tent.

When we artificially made it so no updates were detected as being needed, this segment would process at about 300 RCUs, which was 2.4 MB/sec or about 5,000 items per second. It would take about 19 minutes to read through it.

When updates were needed on every item though, it only performed at about 200 UpdateItem calls per second, probably due to 5ms round trip times on UpdateItem calls. That rate required about 7 hours to process the segment.

There is no `BatchUpdateItem` API call. Without that, to scale up we need to have more client threads making the UpdateItem API call. We could change so instead of having the same worker that's doing the scan do the updates right away, we could have each worker doing scans return the "commands" to be performed, scatter those commands via 'repartition' to the various workers, then run those commands in mass parallel across the workers.

If the items with the same PK were all in the same partition, this would speed things from 200/sec to 1,000/sec (assuming small items, limited only by per-partition write rates). If the PK value had been previously allocated across 10 partitions (as was the case in the above test), this would go from 200/sec (limited by how fast one client thread can loop) to 10,000/sec (limited by how fast the 10 partitions can perform writes).

You'd test it by generating a table with a majority of items having a single PK value and making sure we still get good parallelization for the update calls.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating the bulk-update scan and UpdateItem worker, then trace how work is distributed between them. Done means a heavily skewed table still parallelizes update calls and the focused benchmark confirms improved throughput.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
databases, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.