dandi / dandi/dandi-cli

ls: should not "add up" dandiset size in the total

Open
#583 0 comments 0 reactions 0 assignees View on GitHub
cmd-ls UX
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

ATM
```
$ dandi ls -r -f pyout https://gui-beta-dandiarchive-org.netlify.app/#/dandiset/000003
PATH VERSION NAME ASSET_COUNT SIZE CREATED MODIFIED METADATA ASSET_ID
000003 draft Layer-Specific Physiolog... 22 541.8 GB 2021-04-20T19:12:39.326596Z 2021-04-20T19:39:10.015843Z {'id': 'DANDI:000003/dra...
...es-YutaMouse20-140327_behavior+ecephys.nwb 8.4 GB 2021-04-20T19:17:17.322645Z 2021-04-20T19:17:17.322672Z {'path': 'sub-YutaMouse2... 4112d7b0-071e-4193-bd30...
...es-YutaMouse20-140321_behavior+ecephys.nwb 11.7 GB 2021-04-20T19:17:40.084220Z 2021-04-20T19:17:40.084244Z {'path': 'sub-YutaMouse2... 05a80228-04a7-4c3b-88d3...
...es-YutaMouse20-140325_behavior+ecephys.nwb 17.3 GB 2021-04-20T19:18:08.365632Z 2021-04-20T19:18:08.365721Z {'path': 'sub-YutaMouse2... 006380d2-c5a6-4394-afe9...
...es-YutaMouse20-140324_behavior+ecephys.nwb 15.1 GB 2021-04-20T19:18:12.389179Z 2021-04-20T19:18:12.389283Z {'path': 'sub-YutaMouse2... 948a507c-fd65-4e84-a7e6...
...es-YutaMouse20-140328_behavior+ecephys.nwb 18.4 GB 2021-04-20T19:18:17.409903Z 2021-04-20T19:18:17.409927Z {'path': 'sub-YutaMouse2... 8538db7e-b726-417c-b26d...
...es-YutaMouse33-150218_behavior+ecephys.nwb 11.9 GB 2021-04-20T19:18:25.266585Z 2021-04-20T19:18:25.266610Z {'path': 'sub-YutaMouse3... 8d21b488-9663-4816-8ff5...
...es-YutaMouse33-150219_behavior+ecephys.nwb 19.6 GB 2021-04-20T19:20:18.097346Z 2021-04-20T19:20:18.097395Z {'path': 'sub-YutaMouse3... 388cfd71-e47c-4c0f-b443...
...es-YutaMouse33-150220_behavior+ecephys.nwb 22.9 GB 2021-04-20T19:20:46.561055Z 2021-04-20T19:20:46.561092Z {'path': 'sub-YutaMouse3... 64c77c31-835e-4e97-b221...
...es-YutaMouse33-150221_behavior+ecephys.nwb 28.9 GB 2021-04-20T19:21:46.741366Z 2021-04-20T19:21:46.741395Z {'path': 'sub-YutaMouse3... 0d1ba8c3-bcaa-487a-aaa6...
...es-YutaMouse33-150226_behavior+ecephys.nwb 27.5 GB 2021-04-20T19:21:54.741922Z 2021-04-20T19:21:54.741945Z {'path': 'sub-YutaMouse3... cb67a817-05d0-46fc-9c6e...
...es-YutaMouse37-150609_behavior+ecephys.nwb 5.3 GB 2021-04-20T19:22:32.531087Z 2021-04-20T19:22:32.531117Z {'path': 'sub-YutaMouse3... 60b39cbe-e838-4945-b3c8...
...es-YutaMouse37-150610_behavior+ecephys.nwb 10.1 GB 2021-04-20T19:24:28.079987Z 2021-04-20T19:24:28.080010Z {'path': 'sub-YutaMouse3... dec36ce4-30c8-4eb4-968f...
...es-YutaMouse37-150611_behavior+ecephys.nwb 17.4 GB 2021-04-20T19:28:33.037656Z 2021-04-20T19:28:33.037687Z {'path': 'sub-YutaMouse3... 177ebdc6-755e-428f-bf23...
...es-YutaMouse37-150612_behavior+ecephys.nwb 13.9 GB 2021-04-20T19:31:13.944874Z 2021-04-20T19:31:13.944908Z {'path': 'sub-YutaMouse3... ab6c0d88-4d34-47f0-a0b2...
...es-YutaMouse37-150617_behavior+ecephys.nwb 6.6 GB 2021-04-20T19:32:22.205559Z 2021-04-20T19:32:22.205583Z {'path': 'sub-YutaMouse3... 4704251b-5f10-498b-990c...
...es-YutaMouse33-150225_behavior+ecephys.nwb 97.5 GB 2021-04-20T19:33:09.937688Z 2021-04-20T19:33:09.937727Z {'path': 'sub-YutaMouse3... 572b9802-31c3-41c3-adeb...
...es-YutaMouse39-150727_behavior+ecephys.nwb 11.1 GB 2021-04-20T19:34:57.907438Z 2021-04-20T19:34:57.907486Z {'path': 'sub-YutaMouse3... 76029865-fbad-46e7-91a0...
...es-YutaMouse39-150728_behavior+ecephys.nwb 8.0 GB 2021-04-20T19:36:32.107319Z 2021-04-20T19:36:32.107342Z {'path': 'sub-YutaMouse3... 2104bbb1-684d-4d09-9b43...
...es-YutaMouse38-150709_behavior+ecephys.nwb 18.0 GB 2021-04-20T19:36:46.126210Z 2021-04-20T19:36:46.126233Z {'path': 'sub-YutaMouse3... 5c09d382-a73c-4cf0-9835...
...es-YutaMouse39-150730_behavior+ecephys.nwb 14.7 GB 2021-04-20T19:38:35.775584Z 2021-04-20T19:38:35.775627Z {'path': 'sub-YutaMouse3... 4e4561a5-54b7-4ec7-8c87...
...es-YutaMouse39-150729_behavior+ecephys.nwb 14.3 GB 2021-04-20T19:38:44.288045Z 2021-04-20T19:38:44.288068Z {'path': 'sub-YutaMouse3... d9e1d779-8b11-45df-b523...
...es-YutaMouse33-150223_behavior+ecephys.nwb 143.5 GB 2021-04-20T19:39:09.999447Z 2021-04-20T19:39:09.999471Z {'path': 'sub-YutaMouse3... a202ccba-23be-4ea8-b66f...
Summary: 1.1 TB
```
because I guess dandi-api reports size also for dandiset. And then we add it up to individual assets as well, thus doubling it all up.
I think the fix should be to `pyout` output specific, but I am not sure yet if it should exclude dandiset's record, or better just group by some type of record. Later on we might want to `ls` over dandisets (without `-r`).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `dandi ls -r -f pyout` output path and trace how the dandiset record and individual asset records contribute to the Summary size. Determine whether pyout should exclude the dandiset size or group records by type; done means the summary no longer double-counts the dandiset and its assets, with recursive asset output preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.