gchq / gchq/sleeper

Changes to max.concurrent.tasks instance properties does not take effect

Open
#7,090 2 comments 0 reactions 0 assignees View on GitHub
usability
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### Description / Background

Changed the value of
```
sleeper.bulk.export.max.concurrent.tasks
```
from 300 to 750. 2 hours later and the change has still not been picked up and only a max of 300 containers are running. I assume there's a similar problem with changing `sleeper.ingest.max.concurrent.tasks` and `sleeper.compaction.max.concurrent.tasks` but I've not verified this.

### Steps to reproduce

1. Start a standard ingest / compaction / bulk export
2. Update instance properties so that job can be processed faster
3. The max concurrent containers deployed does not change

### Expected behaviour

The number of concurrent containers increases to the new limit within a small number of minutes

### Technical Notes / Implementation Details

We need to decide whether to use an InstancePropertiesProvider or PropertiesReloader to resolve this. An InstancePropertiesProvider would have similar functionality to TablePropertiesProvider, but doesn't exist yet. PropertiesReloader has very limited functionality.

One disadvantage of PropertiesReloader is that it relies on mutability of InstanceProperties. This means it's probably not applicable to multithreaded contexts, particularly MultiThreadedStateStoreCommitter.

We'd like to use this issue to experiment with an InstancePropertiesProvider, for potential application across the codebase.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how sleeper.bulk.export.max.concurrent.tasks, sleeper.ingest.max.concurrent.tasks, and sleeper.compaction.max.concurrent.tasks are read, then compare the proposed InstancePropertiesProvider with TablePropertiesProvider and PropertiesReloader. Done means updating instance properties changes the corresponding concurrency limits within minutes, with coverage for the affected processing paths and multithreaded safety.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.