Changes to max.concurrent.tasks instance properties does not take effect
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### Description / Background
Changed the value of
```
sleeper.bulk.export.max.concurrent.tasks
```
from 300 to 750. 2 hours later and the change has still not been picked up and only a max of 300 containers are running. I assume there's a similar problem with changing `sleeper.ingest.max.concurrent.tasks` and `sleeper.compaction.max.concurrent.tasks` but I've not verified this.
### Steps to reproduce
1. Start a standard ingest / compaction / bulk export
2. Update instance properties so that job can be processed faster
3. The max concurrent containers deployed does not change
### Expected behaviour
The number of concurrent containers increases to the new limit within a small number of minutes
### Technical Notes / Implementation Details
We need to decide whether to use an InstancePropertiesProvider or PropertiesReloader to resolve this. An InstancePropertiesProvider would have similar functionality to TablePropertiesProvider, but doesn't exist yet. PropertiesReloader has very limited functionality.
One disadvantage of PropertiesReloader is that it relies on mutability of InstanceProperties. This means it's probably not applicable to multithreaded contexts, particularly MultiThreadedStateStoreCommitter.
We'd like to use this issue to experiment with an InstancePropertiesProvider, for potential application across the codebase.
Contributor guide
Research direction
Start by tracing how sleeper.bulk.export.max.concurrent.tasks, sleeper.ingest.max.concurrent.tasks, and sleeper.compaction.max.concurrent.tasks are read, then compare the proposed InstancePropertiesProvider with TablePropertiesProvider and PropertiesReloader. Done means updating instance properties changes the corresponding concurrency limits within minutes, with coverage for the affected processing paths and multithreaded safety.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100