Add repeatable tests for ingesting data at various scales and document cost
Open
enhancement
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
Under epic:
- https://github.com/gchq/sleeper/issues/1391
Add options to the system test module to test ingesting various scales of random data, e.g. 1 billion, 10 billion, 100 billion, 1 trillion, and document the resulting costs.
Note there's a related issue for a performance test specific to standard ingest: https://github.com/gchq/sleeper/issues/25
We would like to test this for both standard ingest and bulk import, up to moderate scales for standard ingest and all scales for bulk import.
Contributor guide
Assessment
This issue has not been assessed yet.