SpikeInterface / SpikeInterface/spikeinterface
impact of n_jobs and chunk_size on writing speed
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 847
- Forks
- 280
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 29
Description
I wanted to test how n_jobs and chunk_size affects writing speed, since the behavior didn't seem linear to me.
I ran a series of tests with a computer with 64 cores and 128GB of RAM running Linux (Linux-5.4.0-177-generic-x86_64-with-glibc2.31). The test consists in writing a 31.54GB file (384 channels - 30.0kHz - 1 segments - 44,096,989 samples).
the command I used was:
si.write_binary_recording(recording=rec,
file_paths=str(target.resolve()),
add_file_extension=False,
dtype="int16",
verbose=True,
n_jobs=n_jobs,
chunk_size=chunk_size,
progress_bar=True)
The results are summarized in the graph below, and clearly show that there write times initially decrease quickly with adding more CPUs, but the effect then dissipate, probably because of increase overhead past ~24 CPUs. We can also observe how chunk_size does not have a huge impact, as long as it is large enough (>= 30,000).
(Each line is the average of 3 trials)
The fastest average write time was obtained for
n_jobs chunk_size time
24 651000 28.512982
I'm sure there are tons of factor that influence the write time, but maybe this could be informative for your users.
The full results are attached
Benchmark_results.csv
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the write_binary_recording call shown in the issue and review the attached Benchmark_results.csv alongside the reported hardware and workload. The issue does not identify a file, test, or specific change; done would require deciding whether these results should become user-facing guidance or documentation and defining the expected content.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100