gchq / gchq/sleeper

Add repeatable tests for ingesting data at various scales and document cost

Open
#28 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

Under epic:
- https://github.com/gchq/sleeper/issues/1391

Add options to the system test module to test ingesting various scales of random data, e.g. 1 billion, 10 billion, 100 billion, 1 trillion, and document the resulting costs.

Note there's a related issue for a performance test specific to standard ingest: https://github.com/gchq/sleeper/issues/25

We would like to test this for both standard ingest and bulk import, up to moderate scales for standard ingest and all scales for bulk import.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.