Uploading exported data to s3 without storing at the local disk
- Dominant language
- Java
- Stars
- 17
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
Copying [#309](https://github.com/awslabs/amazon-neptune-tools/issues/309) from @SergeVil from the old amazon-neptune-tools repository.
We are running the Neptune AWS Batch Fargate that has a local disk limitation 20Gb, while the memory is pretty big - 128Gb. Such a low disk volume limits us to export the large database (getting the OS error). We look for the option to upload the exported csv files right away from JVM memory to the S3 bucket. Thank you!
Contributor guide
Research direction
The issue names no files, tests, or entry points. Begin by locating where the Java export process writes CSV data to local disk and how S3 uploads are handled. Done means exported data can reach S3 without exceeding the 20 GB local-disk limit, with coverage for large exports.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100