SpikeInterface / SpikeInterface/spikeinterface

impact of n_jobs and chunk_size on writing speed

Open
#3,252 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Python
Stars
847
Forks
280
Avg merge
3d 9h
Merged PRs (30d)
29

Description

I wanted to test how n_jobs and chunk_size affects writing speed, since the behavior didn't seem linear to me.

I ran a series of tests with a computer with 64 cores and 128GB of RAM running Linux (Linux-5.4.0-177-generic-x86_64-with-glibc2.31). The test consists in writing a 31.54GB file (384 channels - 30.0kHz - 1 segments - 44,096,989 samples).
the command I used was:

si.write_binary_recording(recording=rec, 
                              file_paths=str(target.resolve()), 
                              add_file_extension=False, 
                              dtype="int16", 
                              verbose=True, 
                              n_jobs=n_jobs, 
                              chunk_size=chunk_size, 
                              progress_bar=True)

The results are summarized in the graph below, and clearly show that there write times initially decrease quickly with adding more CPUs, but the effect then dissipate, probably because of increase overhead past ~24 CPUs. We can also observe how chunk_size does not have a huge impact, as long as it is large enough (>= 30,000).
Figure 9

(Each line is the average of 3 trials)

The fastest average write time was obtained for

 		
n_jobs 	chunk_size 	time
24 	651000 	        28.512982

I'm sure there are tons of factor that influence the write time, but maybe this could be informative for your users.

The full results are attached
Benchmark_results.csv

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the write_binary_recording call shown in the issue and review the attached Benchmark_results.csv alongside the reported hardware and workload. The issue does not identify a file, test, or specific change; done would require deciding whether these results should become user-facing guidance or documentation and defining the expected content.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.