Slow merging of spike output files

Hi. I noticed in some of my simulations of a large scale network set up in bmtk that a substantial amount of the overall simulation time for a job is spent reading in and converting temporary spike files from each MPI rank (w. MPI pool size 1536). The issue at hand seems to be due to the serial function _merge_files at https://github.com/AllenInstitute/bmtk/blob/develop/bmtk/utils/io/spike_trains.py#L116, where cvs.reader(...) returns an iterable over each line in the input files. Reading in files as typed arrays and sorting and dumping everything in the end is many times faster as the double for loop containing a function call to write stuff is avoided; something like:

import numpy as np
import h5py
from glob import glob
import os

outputdir = 'output'
dtype=[('timestamps', 'f8'), ('gids', 'u8')]

spikes = np.empty(shape=(), dtype=dtype)
for fname in glob(os.path.join(outputdir, '_bmtk_tmp_spikes_*.csv')):
    spikes = np.r_[spikes, np.loadtxt(fname, delimiter=' ', dtype=dtype)]

sort = np.argsort(spikes['timestamps'])
spikes  = spikes[sort]

np.savetxt(os.path.join(outputdir, 'spikes.csv'), spikes)

f = h5py.File(os.path.join(outputdir, 'spikes.h5'), 'w')
grp = f.create_group('spikes')
grp.attrs['sorting'] = 'time'
grp['gids'] = spikes['gids']
grp['timestamps'] = spikes['timestamps']
f.close()

Any plans on addressing this issue? In my case with somewhat long simulation durations (20s biological time) the utilization of resources is poor as 1 process converts files while 1535 other processes are dormant for 1-2 hours before it to finishes.

Start with bmtk/utils/io/spike_trains.py at _merge_files, especially the CSV reader and output-writing loop described in the issue. Compare the current merge path with the proposed NumPy-based loading, sorting, and CSV/HDF5 output, then benchmark large sets of temporary rank files to confirm the conversion time improves without changing the generated spike files.