Incremental writes and hotspotting
- Dominant language
- Go
- Stars
- 187
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
Mteigers writes:
> I think S3 still has the recommendation against writing atomically incrementing files as they are served by the same compute cluster and at scale can cause hotspotting. Maybe they no longer have that advice, my S3 knowledge is a little outdated.
>
> But an alternate would be to provide a fast hash of your counter as a prefix and use like the first 3 characters and use them as “folders”. Something like Meow Hash is supposed to be fast at this so you end up with something like:
>
> meow(000001) = DDB147 (example) And then you write DDB/000001
https://old.reddit.com/r/golang/comments/1h48ujv/building_a_distributed_log_using_s3_under_150/lzwmjzw/
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are identified. Start by reading the issue discussion and linked Reddit comment, then determine whether the proposed incremental-write hotspotting concern still applies and define an agreed implementation scope; done requires a decided approach.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100