Make use of data more by devising subsampling
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 300
- PR merge metrics
- No merged PRs in 30d
Description
When we use a certain `memory_allocation` [^1] in `subsampling`, we reduce the number of samples until we reach the memory limit.
However, we need to come up with an appropriate value for this as when we set it too high, the training fails due to memory error while when we set it too low, we waste memory.
For now, we circumvent this issue by measuring the memory consumption when using the default config.
[^1]: The definition of the `memory_allocation` is the following:
Absolute memory in MB, e.g. 10MB is ``"memory_allocation": 10``.
The memory used by the dataset is checked after each reduction method is performed.
If the dataset fits into the allocated memory, any further methods listed in ``"methods"`` will not be performed.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.