apache / apache/accumulo-testing
Explore storing continuous ingest bulk import files in S3
Open
- Dominant language
- Java
- Stars
- 19
- Forks
- 40
- Avg merge
- 21h 30m
- Merged PRs (30d)
- 1
Description
When running bulk import continuous ingest test it can take a while to generate a good bit of data to start testing. Not sure, but it may be faster to generate a data set once and store it in S3. Then future test could possibly use that data set.
I think it would be interesting to experiment with this and if it works well add documentation to the bulk import test docs explaining how to do it. One gotcha with this approach is that anyone running a test needs to be consistent with split points. A simple way to address this problem would be store a file of split points in S3 with the data.
Contributor guide
Assessment
This issue has not been assessed yet.