VH-Lab / VH-Lab/Pyraview

[Optional — design suggestion, not a bug] Folder scan merges every `*_L*.bin` in a directory into one dataset

Open
#29 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement optional question
Dominant language
MATLAB
Stars
0
Forks
0
Avg merge
30m
Merged PRs (30d)
6

Description

This is not a bug report. The current behaviour is self-consistent, matches the MATLAB binding, and is correct under the convention of one dataset per folder. This is a suggestion about whether that convention should be enforced rather than assumed. Closing it as "working as intended" is a perfectly reasonable outcome.

Current behaviour

PyraviewDataset._scan_folder (and MATLAB's Dataset.scanFolder) selects level files with a wildcard prefix:

for full_path in sorted(glob.glob(os.path.join(self.folder_path, '*_L*.bin'))):
d = dir(fullfile(obj.FolderPath, '*_L*.bin'));

Nothing associates expA_L1.bin with expA_L2.bin as belonging to the same recording — the prefix is a wildcard, so every pyramid file in the directory is treated as a level of a single dataset.

What that looks like with two recordings in one folder

expA at 1000 Hz starting at 5 s, expB at 500 Hz starting at 900 s, each with two levels:

files on disk : ['expA_L1.bin', 'expA_L2.bin', 'expB_L1.bin', 'expB_L2.bin']

native_rate   : 1000.0   (expA is 1000, expB is 500)
start_time    : 5.0      (expA is 5.0, expB is 900.0)
files         : ['expA_L1.bin', 'expB_L1.bin', 'expA_L2.bin', 'expB_L2.bin']
decimations   : [10, 10, 100, 100]
rates         : [100.0, 50.0, 10.0, 5.0]
start times   : [5.0, 900.0, 5.0, 900.0]

get_data(5..15s, 50px) -> (0,)   | level chosen from: expB_L2.bin

Three consequences, all silent:

  1. The two recordings merge into one pyramid, with duplicate decimation factors (10, 10, 100, 100) sorted together as if they were levels of one dataset.
  2. native_rate, native_start_time, channels and data_type come from whichever file sorts first; the other recording's metadata is discarded.
  3. Level selection can cross datasets. Asking for expA's window selects expB_L2.bin, because 5 Hz is the coarsest rate meeting the demand — and expB starts at 900 s, so the read falls outside the file and returns empty.

Why this may well be fine as-is

  • The MATLAB binding behaves identically. Changing only Python would reintroduce exactly the kind of divergence #22 closed, so any change should land in both.
  • One dataset per folder is a reasonable convention, and may be the only layout in practice. The test suites assume it (each test writes one dataset into a fresh temp directory), which is realistic rather than an oversight.
  • The property-based constructor already sidesteps it. Passing files=[...] explicitly — the NDI use case — never consults the glob, so the ambiguity does not arise there.

Options, if it seems worth addressing

  1. An optional prefix argument. PyraviewDataset(folder, prefix='expA') → globs expA_L*.bin; MATLAB Dataset(folder, 'Prefix', "expA"). Backward compatible, makes intent explicit.
  2. A guard. Warn or raise when the scan finds more than one distinct filename prefix, or duplicate decimation factors. Cheap, and turns a silent wrong answer into a clear message. Could be done without option 1.
  3. Documentation only. State the one-dataset-per-folder assumption in docs/API.md and the docstrings, and leave the behaviour alone.

My suggestion would be 2 on its own, or 1+2 together — 2 is what converts the failure mode from "quietly returns the wrong data" into something a user can act on. But this is a judgement call about how the library is actually used, which is yours to make.

Noticed while verifying the v0.4.1 wheels: a scratch directory held pyramid files from two earlier checks, and get_data returned an empty array rather than the expected samples.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Python's PyraviewDataset._scan_folder and MATLAB's Dataset.scanFolder, then review the existing tests and docs/API.md mentioned in the issue. The maintainer must first choose between a prefix, a guard, or documentation-only approach; done means the decision is implemented consistently in both bindings or documented clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
matlab, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.