AllenNeuralDynamics / AllenNeuralDynamics/aind-ephys-transformation

Zarr shape mismatch error

Open
#46 12 comments 0 reactions 1 assignee Claimed by @alejoe91 View on GitHub
Dominant language
Python
Stars
0
Forks
1
PR merge metrics
No merged PRs in 30d

Description

I received the following error when uploading ephys data from VAST to S3. The odd thing is the upload was successful after retrying, indicating that this may be a bug with how the zarr container is initialized.

Error Trace:
```
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.10/site-packages/aind_ephys_transformation/ephys_job.py", line 1138, in
job_response = job.run_job()
File "/usr/local/lib/python3.10/site-packages/aind_ephys_transformation/ephys_job.py", line 1115, in run_job
self._compress_raw_data()
File "/usr/local/lib/python3.10/site-packages/aind_ephys_transformation/ephys_job.py", line 1091, in _compress_raw_data
self._compress_and_write_block(
File "/usr/local/lib/python3.10/site-packages/aind_ephys_transformation/ephys_job.py", line 1015, in _compress_and_write_block
write_or_append_recording_to_zarr(
File "/usr/local/lib/python3.10/site-packages/aind_ephys_transformation/compression_utils.py", line 38, in write_or_append_recording_to_zarr
add_or_append_recording_to_zarr_group(
File "/usr/local/lib/python3.10/site-packages/aind_ephys_transformation/compression_utils.py", line 96, in add_or_append_recording_to_zarr_group
add_or_append_traces_to_zarr(
File "/usr/local/lib/python3.10/site-packages/aind_ephys_transformation/compression_utils.py", line 266, in add_or_append_traces_to_zarr
executor.run(recording_slices=recording_slices)
File "/usr/local/lib/python3.10/site-packages/spikeinterface/core/job_tools.py", line 500, in run
res = self.func(segment_index, frame_start, frame_stop, worker_dict)
File "/usr/local/lib/python3.10/site-packages/aind_ephys_transformation/compression_utils.py", line 360, in _write_zarr_chunk_append
zarr_dataset[start_frame_global:end_frame_global, :] = traces
File "/usr/local/lib/python3.10/site-packages/zarr/core.py", line 1447, in __setitem__
self.set_orthogonal_selection(pure_selection, value, fields=fields)
File "/usr/local/lib/python3.10/site-packages/zarr/core.py", line 1636, in set_orthogonal_selection
self._set_selection(indexer, value, fields=fields)
File "/usr/local/lib/python3.10/site-packages/zarr/core.py", line 1962, in _set_selection
check_array_shape("value", value, sel_shape)
File "/usr/local/lib/python3.10/site-packages/zarr/util.py", line 561, in check_array_shape
raise ValueError(
ValueError: parameter 'value': expected array with shape (0, 384), got (16320, 384)
NoneType: None
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.