ContextLab / ContextLab/clustrix
parallel=True: the returned shape depends on cores and on the loop length, and what it should be is undecided
- Dominant language
- Python
- Stars
- 10
- Forks
- 4
- Avg merge
- 6h 27m
- Merged PRs (30d)
- 9
Description
## What
For a scalar-returning callee, `@cluster(parallel=True, ...)` returns a
different answer depending on `cores`, and a different *type* depending on how
many times the loop runs. Neither is documented as a promise; both are
observable today on the local path.
Reproduction (real, run on this branch):
```python
import clustrix
clustrix.configure(cluster_type="local", cluster_host=None)
def partial_sum(n, _parallel_i=None):
indices = list(range(n)) if _parallel_i is None else list(_parallel_i)
marker = 0
for i in range(n):
marker = i * i
del marker
return sum(indices)
```
```text
partial_sum(8) -> 28
@cluster(parallel=True, cores=1) partial_sum(8) -> [6, 22]
@cluster(parallel=True, cores=2) partial_sum(8) -> [1, 5, 9, 13]
@cluster(parallel=True, cores=4) partial_sum(8) -> [0, 1, 2, 3, 4, 5, 6, 7]
@cluster(parallel=True, cores=2) partial_sum(2) -> 1 (an int)
```
## Why this is a design question and not simply a bug
`_combine_local_results` (`clustrix/decorator.py`) has four branches:
```python
if not results: return None
if len(results) == 1: return results[0]
if all(isinstance(r, list) for r in results): ...concatenate...
return results
```
* For a **list-returning** callee the concatenation is right: a parallel run
reproduces the sequential answer exactly, whatever the chunk count.
* For a **scalar-returning** callee no branch can be right. One chunk returns
the complete answer; two chunks return two partial answers that only the
caller knows how to reduce. Unwrapping the single result is therefore not a
bug on its own -- it is the correct answer for the one-chunk case -- and
wrapping it would be equally defensible. The inconsistency is that the two
cases disagree, and fixing that means deciding what `parallel=True` promises:
1. always a list of per-chunk answers (consistent, breaks the one-chunk case
and every list-returning caller who relies on concatenation);
2. accept a reducer from the caller (`@cluster(parallel=True, reduce=sum)`),
which is the only option that can produce a *correct* scalar;
3. refuse to parallelize a callee whose answer is not a list, and say so.
Changing any of this is user-visible and needs its own release note, which is
why #152's round four pinned the current behaviour instead of altering it.
## The short-loop type change
`LoopInfo._assess_parallelizability` requires `iteration_count >= 3`, so a
loop of one or two iterations is not split at all and the caller gets the
sequential scalar. A caller who tested on a short input and shipped a long one
sees an `int` become a `list`. Whatever is decided above should say something
about this boundary too.
## The unreachable branch
`if len(results) == 1: return results[0]` cannot be reached from the decorator
today. Splitting requires at least three iterations, and
`chunk_size = max(1, len(loop_range) // (workers * 2))` cuts any loop of three
or more into at least two chunks. Verified exhaustively for every range length
0..199 against every worker count 1..64: no combination yields a single chunk.
It is currently a contract for the helper's callers rather than live code, and
whatever is decided above should either make it reachable or remove it.
## Current state
* Behaviour is pinned by
`tests/unit/test_local_cores.py::test_the_answers_shape_depends_on_cores_and_on_how_long_the_loop_is`
and by `tests/test_decorator.py::TestResultCombination::test_combine_local_results_single`.
* Documented in `docs/source/limitations.rst` under "Parallel and sequential
runs can return different shapes".
Follow-up from #152.
Contributor guide
Research direction
Read clustrix/decorator.py's _combine_local_results and LoopInfo._assess_parallelizability, then run the pinned cases in tests/unit/test_local_cores.py::test_the_answers_shape_depends_on_cores_and_on_how_long_the_loop_is and tests/test_decorator.py::TestResultCombination::test_combine_local_results_single. Review docs/source/limitations.rst and follow-up #152 before deciding the promised result shape, reducer behavior, and short-loop boundary; done means the decision is reflected consistently in tests, documentation, and the release note.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100