assetsSummary undercounts numberOfSubjects/numberOfFiles after replacing a single asset
- Dominant language
- Python
- Stars
- 26
- Forks
- 21
- Avg merge
- 4d 23h
- Merged PRs (30d)
- 15
Description
## Evidence / live problem
Sandbox dandiset [218201 draft](https://sandbox.dandiarchive.org/dandiset/218201/draft) shows **Number of Subjects: 17** in the asset summary, but there are 18 subjects. This dandiset originally had 18 subjects; one subject's file (`sub-mouse-12`) was later replaced.
From the sandbox API:
- `.../versions/draft/assets/?page_size=100` → **18 assets**, one per subject (`sub-mouse-1` … `sub-mouse-18`), each with a distinct participant identifier (`mouse_1` … `mouse_18`). The replaced asset (`sub-mouse-12`) is the most recently created.
- `.../versions/draft/info/` → version `status: "Valid"`, `asset_validation_errors: []` — i.e. all 18 assets appear to be currently `VALID`.
- Same response → `assetsSummary.numberOfFiles: 17` and `assetsSummary.numberOfSubjects: 17`.
So the summary reflects 17 valid assets even though 18 are valid, and `sub-mouse-12` (the replacement) is the one missing from the counts. The undercounted `numberOfSubjects` seems to be a symptom of the whole `assetsSummary` being stale rather than a subject-specific problem.
## Claude's hypothesis
It seems that a draft version's `assetsSummary` can get stuck undercounting: it may be computed at a moment when a just-added/replaced asset is not yet `VALID`, and then possibly never recomputed after that asset becomes `VALID`.
A plausible chain, based on a read of the code:
1. Aggregation counts only `VALID` assets — `version_aggregate_assets_summary` in `dandiapi/api/services/metadata/__init__.py` aggregates `version.assets.filter(status=Asset.Status.VALID)`.
2. Aggregation appears to be dispatched only for versions in `PENDING` — `validate_draft_version_metadata` in `dandiapi/api/tasks/scheduled.py` selects `status == PENDING` versions and `.delay()`s both `validate_version_metadata_task` and `aggregate_assets_summary_task` as independent, unordered Celery tasks.
3. Replacing/adding an asset sets the version to `PENDING` and creates a new `PENDING` asset (`dandiapi/api/services/asset/__init__.py`).
4. On the next scheduled cycle, asset validation and version aggregation seem to race. If aggregation runs before the new asset flips to `VALID`, it would write a summary that omits it.
5. When the asset later becomes `VALID`, aggregation may not be re-triggered — `validate_asset_metadata` only bumps the draft version's `modified` timestamp (not its status), while `validate_version_metadata` has already moved the version out of `PENDING` (`VALIDATING` → `VALID`), so later scheduled cycles would skip re-dispatching aggregation.
If that's what's happening, the summary would be frozen at the value computed before the replacement asset finished validating. This would also be consistent with the original 18-subject upload being correct while a single-file replacement is not.
## Possible directions (to confirm after tracing)
- Re-trigger aggregation when an asset finishes validating — e.g. in `validate_asset_metadata`, put the owning draft version(s) back into `Status.PENDING` on the success path so the next scheduled sweep re-runs aggregation. (Would need to confirm this doesn't cause a re-validation loop; `validate_pending_asset_metadata` appears to key off `Asset.status == PENDING`, not version status.)
- Or dispatch `aggregate_assets_summary_task` for the affected draft versions directly from the asset-validation completion path, which would also avoid transient undercounts.
Possibly related: #2091 ("Finalizing a zarr should put dandiset in `PENDING` state"), which looks like the same class of "downstream completion doesn't re-trigger the `PENDING` → aggregate cycle" issue.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing version_aggregate_assets_summary in dandiapi/api/services/metadata/__init__.py alongside validate_draft_version_metadata and the validation tasks in dandiapi/api/tasks/scheduled.py. Then inspect asset replacement and validation in dandiapi/api/services/asset/__init__.py; done means confirming the summary is recomputed after a replacement asset becomes VALID without creating a validation loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100