avinassh / avinassh/s3-log

Incremental writes and hotspotting

Open
#7 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
187
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Mteigers writes:

> I think S3 still has the recommendation against writing atomically incrementing files as they are served by the same compute cluster and at scale can cause hotspotting. Maybe they no longer have that advice, my S3 knowledge is a little outdated.
>
> But an alternate would be to provide a fast hash of your counter as a prefix and use like the first 3 characters and use them as “folders”. Something like Meow Hash is supposed to be fast at this so you end up with something like:
>
> meow(000001) = DDB147 (example) And then you write DDB/000001

https://old.reddit.com/r/golang/comments/1h48ujv/building_a_distributed_log_using_s3_under_150/lzwmjzw/

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are identified. Start by reading the issue discussion and linked Reddit comment, then determine whether the proposed incremental-write hotspotting concern still applies and define an agreed implementation scope; done requires a decided approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.