dandi / dandi/dandi-cli

pyout might be dragging us behind on large dandisets

Open
#1,549 1 comment 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

While troubleshooting with @Kevancic the download of 000026 at MGH we observed that process is at 100% while not downloading much

py-spy top gave following

![image](https://github.com/user-attachments/assets/132b5326-32ed-4cf3-89f6-eb3a0d488495)

invocation was

```
DANDI_DEVEL_AGGRESSIVE_RETRY=1 dandi --log-level=INFO download -e refresh -J 1:1 https://dandiarchive.org/dandiset/000026/draft/files?location=sub-I48
```

note: we still were logging at DEBUG level in our log file, there were no pyout (?) logs there, but still were spending time validating using jsonschema every bloody record...

In general it should not be a problem but I think it is that within long running process and some "summary" or alike events start to pile up.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.