fieldtrip / fieldtrip/fieldtrip

resampledesign: reported "maximum number of unique permutations" is k^n, but the generator draws from (k!)^n for >2 repeated measures

Open
#2,615 0 comments 0 reactions 0 assignees View on GitHub
enhancement question unconfirmed
Dominant language
MATLAB
Stars
987
Forks
768
Avg merge
2d 14h
Merged PRs (30d)
1

Description

In `private/resampledesign.m`, the number of unique permutations reported for a
within-unit (repeated measures) design and the number the resampling loop can
actually produce appear to disagree whenever a unit has more than two
observations. I may well be missing something, so I would rather ask than
assume.

`unitlen(i)` is the number of observations per unit:

```matlab
unitsel{i} = find(all(design(cfg.uvar,:)==repmat(unitlevel(:,i), 1, Nrepl), 1));
unitlen(i) = length(unitsel{i});
```

and the maximum is reported, and used to warn, as the product of those lengths:

```matlab
fprintf('the maximum number of unique permutations is %d\n', prod(unitlen));

if cfg.numrandomization > prod(unitlen)
ft_warning('the number of randomizations (%d) is larger than the maximum number of unique permutations (%d), ...')
```

But the permutation itself is a full reordering within each unit:

```matlab
for i=1:cfg.numrandomization
for j=1:length(unitlevel)
resample(i, unitsel{j}) = unitsel{j}(randperm(length(unitsel{j})));
end
end
```

`randperm(k)` yields all `k!` orderings per unit, so the space the loop samples
from is `prod(factorial(unitlen))`, not `prod(unitlen)`. The two agree only when
every unit has exactly two observations, since `2! = 2`. That is consistent with
the adjacent restriction:

```matlab
if any(unitlen~=2)
ft_error('cfg.numrandomization=''all'' is only supported for two repeated measurements');
end
```

**Concrete case.** A within-subject design with 3 conditions and 12 participants:

- reported maximum: `3^12` = 531,441
- orderings the loop can generate: `6^12` = 2,176,782,336

so `cfg.numrandomization = 10000` is 1.9% of the true space but is reported
against a number ~4000x smaller.

**Practical consequence.** With 3+ conditions the warning fires much earlier than
the sampling actually warrants, which can lead users to reduce
`cfg.numrandomization` unnecessarily and lose p-value resolution (the floor is
`1/(numrandomization+1)`). We hit exactly this in our own pipeline, where a guard
mirroring `prod(unitlen)` capped the randomization count for small samples.

**Question.** Is `prod(unitlen)` intended as a deliberately conservative bound, or
does "unique permutations" here mean something narrower than the set of
orderings `randperm` produces? If it is meant to match the generator,
`prod(factorial(unitlen))` would appear to be the corresponding count.

MATLAB R2025b (behaviour is in the source, not version-specific), FieldTrip
master as of today.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.