dandi / dandi/dandi-cli

interleave::get is "expensive"

Open
#1,551 1 comment 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

Part of troubleshooting of
- #1549

running a straightforward download

```
rm -rf sub-*; DANDI_DEVEL_AGGRESSIVE_RETRY=1 dandi --log-level=INFO download -e refresh -J 1:1 https://api.dandiarchive.org/api/assets/f0da6521-b9c7-4fa5-9405-c48527aebe90/download/
```
which is a single file/thread download leads to dandi CLI being 10-20% CPU and py-spy showing

![image](https://github.com/user-attachments/assets/bdaa7d79-d0a0-4f10-b945-e6094980b23a)

so sampling profiler finds itself most often in `get (interleave/__init__.py)`. There are multiple `get`s in that file but this particular Interleaver.get https://github.com/jwodder/interleave/blob/master/src/interleave/__init__.py#L348 is the primary suspect due to internal `while True` loop.

@jwodder could you please see if code there potentially be made more efficient?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.