aws-samples / aws-samples/eks-spark-benchmark
Consistent store / ListBucket operations
Open
- Dominant language
- Scala
- Stars
- 87
- Forks
- 30
- PR merge metrics
- No merged PRs in 30d
Description
> Limitations: It is suggested to use a consistent store with staging committers together.
from [here](https://github.com/aws-samples/eks-spark-benchmark/blob/master/performance/s3.md#staging-committers).
Can you elaborate more about that? Isn't S3 has [Strong Consistency](https://aws.amazon.com/s3/consistency/)?
From my AWS bills I see that we're doing a lot of `ListBucket` requests. I wonder if the default committer is responsible for that under the hood.
Do I have to use either EFS (for staging committer) or DynamoDB (for magic committer)?
Contributor guide
Assessment
This issue has not been assessed yet.