fluent / fluent/fluent-plugin-s3

S3 cost optimisation: Remember the last index value that was used

Open
#160 17 comments 9 reactions 0 assignees View on GitHub
bug
Dominant language
Ruby
Stars
322
Forks
217
Avg merge
14h 34m
Merged PRs (30d)
5

Description

## The problem

Suppose you use the defaults:
- `s3_object_key_format` of `%{path}%{time_slice}_%{index}.%{file_extension}`
- `time_slice_format` of `%Y%m%d%H`

And suppose you flush every 30 seconds. So 120 files per hour.

The first file will check whether `foo-2016010109_1.gz` exists via an S3 HEAD request, see it doesn't exist, and then upload to that filename.

The next file will be first check whether `foo-2016010109_1.gz` exists via an S3 HEAD request, see it exists, and so increment the index to `foo-2016010109_2.gz`, check whether it exists via with an S3 HEAD request, see it doesn't exist, and then upload to the filename.

This will continue. When we get to the final file of the hour (the 120th file), we'll first do 119 HEAD requests!

That's 1+2+...+119 = 7140 S3 requests over the hour. And that's per input log file, per instance.

S3 HEAD requests are "$0.004 per 10,000 requests". So the monthly cost of the above, for 5 log files for 100 instances amounts to 7140_5_100_24_30*$0.004/1000 = $1028

More generally, 1+2+...+n is O(n^2) and we can reduce this to O(n).
## Solutions

(a) The user can modify `time_slice_format` to include `%M`. Or the default could include `%M`.
(b) fluent-plugin-s3 could remember the last index it uploaded to, and so not have to check whether the n-1 earlier files already exist: fluentd would know they do.

If either solution was implemented, we'd've reduced the number of HEAD requests from O(n^2) to O(n). (Technically (a) doesn't reduce the solution to O(n^2), it just makes our n tiny.)

So rather than 7140 S3 requests per hour per log file per instance, we'd only do 120.

This reduces the monthly cost from $1028 to $17.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the S3 upload path and the HEAD existence checks associated with `s3_object_key_format` and `time_slice_format`. Compare the two proposed solutions, then define how the last index is retained for each time slice and verify that the request count changes from repeated earlier checks to one check per file.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, ruby
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.