[Optional — design suggestion, not a bug] Folder scan merges every `*_L*.bin` in a directory into one dataset
Nobody has claimed this yet.
- Dominant language
- MATLAB
- Stars
- 0
- Forks
- 0
- Avg merge
- 30m
- Merged PRs (30d)
- 6
Description
This is not a bug report. The current behaviour is self-consistent, matches the MATLAB binding, and is correct under the convention of one dataset per folder. This is a suggestion about whether that convention should be enforced rather than assumed. Closing it as "working as intended" is a perfectly reasonable outcome.
Current behaviour
PyraviewDataset._scan_folder (and MATLAB's Dataset.scanFolder) selects level files with a wildcard prefix:
for full_path in sorted(glob.glob(os.path.join(self.folder_path, '*_L*.bin'))):
d = dir(fullfile(obj.FolderPath, '*_L*.bin'));
Nothing associates expA_L1.bin with expA_L2.bin as belonging to the same recording — the prefix is a wildcard, so every pyramid file in the directory is treated as a level of a single dataset.
What that looks like with two recordings in one folder
expA at 1000 Hz starting at 5 s, expB at 500 Hz starting at 900 s, each with two levels:
files on disk : ['expA_L1.bin', 'expA_L2.bin', 'expB_L1.bin', 'expB_L2.bin']
native_rate : 1000.0 (expA is 1000, expB is 500)
start_time : 5.0 (expA is 5.0, expB is 900.0)
files : ['expA_L1.bin', 'expB_L1.bin', 'expA_L2.bin', 'expB_L2.bin']
decimations : [10, 10, 100, 100]
rates : [100.0, 50.0, 10.0, 5.0]
start times : [5.0, 900.0, 5.0, 900.0]
get_data(5..15s, 50px) -> (0,) | level chosen from: expB_L2.bin
Three consequences, all silent:
- The two recordings merge into one pyramid, with duplicate decimation factors (10, 10, 100, 100) sorted together as if they were levels of one dataset.
native_rate,native_start_time,channelsanddata_typecome from whichever file sorts first; the other recording's metadata is discarded.- Level selection can cross datasets. Asking for expA's window selects
expB_L2.bin, because 5 Hz is the coarsest rate meeting the demand — and expB starts at 900 s, so the read falls outside the file and returns empty.
Why this may well be fine as-is
- The MATLAB binding behaves identically. Changing only Python would reintroduce exactly the kind of divergence #22 closed, so any change should land in both.
- One dataset per folder is a reasonable convention, and may be the only layout in practice. The test suites assume it (each test writes one dataset into a fresh temp directory), which is realistic rather than an oversight.
- The property-based constructor already sidesteps it. Passing
files=[...]explicitly — the NDI use case — never consults the glob, so the ambiguity does not arise there.
Options, if it seems worth addressing
- An optional prefix argument.
PyraviewDataset(folder, prefix='expA')→ globsexpA_L*.bin; MATLABDataset(folder, 'Prefix', "expA"). Backward compatible, makes intent explicit. - A guard. Warn or raise when the scan finds more than one distinct filename prefix, or duplicate decimation factors. Cheap, and turns a silent wrong answer into a clear message. Could be done without option 1.
- Documentation only. State the one-dataset-per-folder assumption in
docs/API.mdand the docstrings, and leave the behaviour alone.
My suggestion would be 2 on its own, or 1+2 together — 2 is what converts the failure mode from "quietly returns the wrong data" into something a user can act on. But this is a judgement call about how the library is actually used, which is yours to make.
Noticed while verifying the v0.4.1 wheels: a scratch directory held pyramid files from two earlier checks, and get_data returned an empty array rather than the expected samples.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Python's PyraviewDataset._scan_folder and MATLAB's Dataset.scanFolder, then review the existing tests and docs/API.md mentioned in the issue. The maintainer must first choose between a prefix, a guard, or documentation-only approach; done means the decision is implemented consistently in both bindings or documented clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- matlab, python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100