dandi / dandi/dandi-archive

assetsSummary undercounts numberOfSubjects/numberOfFiles after replacing a single asset

Open
#2,867 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
26
Forks
21
Avg merge
4d 23h
Merged PRs (30d)
15

Description

## Evidence / live problem

Sandbox dandiset [218201 draft](https://sandbox.dandiarchive.org/dandiset/218201/draft) shows **Number of Subjects: 17** in the asset summary, but there are 18 subjects. This dandiset originally had 18 subjects; one subject's file (`sub-mouse-12`) was later replaced.

From the sandbox API:

- `.../versions/draft/assets/?page_size=100` → **18 assets**, one per subject (`sub-mouse-1` … `sub-mouse-18`), each with a distinct participant identifier (`mouse_1` … `mouse_18`). The replaced asset (`sub-mouse-12`) is the most recently created.
- `.../versions/draft/info/` → version `status: "Valid"`, `asset_validation_errors: []` — i.e. all 18 assets appear to be currently `VALID`.
- Same response → `assetsSummary.numberOfFiles: 17` and `assetsSummary.numberOfSubjects: 17`.

So the summary reflects 17 valid assets even though 18 are valid, and `sub-mouse-12` (the replacement) is the one missing from the counts. The undercounted `numberOfSubjects` seems to be a symptom of the whole `assetsSummary` being stale rather than a subject-specific problem.

## Claude's hypothesis

It seems that a draft version's `assetsSummary` can get stuck undercounting: it may be computed at a moment when a just-added/replaced asset is not yet `VALID`, and then possibly never recomputed after that asset becomes `VALID`.

A plausible chain, based on a read of the code:

1. Aggregation counts only `VALID` assets — `version_aggregate_assets_summary` in `dandiapi/api/services/metadata/__init__.py` aggregates `version.assets.filter(status=Asset.Status.VALID)`.
2. Aggregation appears to be dispatched only for versions in `PENDING` — `validate_draft_version_metadata` in `dandiapi/api/tasks/scheduled.py` selects `status == PENDING` versions and `.delay()`s both `validate_version_metadata_task` and `aggregate_assets_summary_task` as independent, unordered Celery tasks.
3. Replacing/adding an asset sets the version to `PENDING` and creates a new `PENDING` asset (`dandiapi/api/services/asset/__init__.py`).
4. On the next scheduled cycle, asset validation and version aggregation seem to race. If aggregation runs before the new asset flips to `VALID`, it would write a summary that omits it.
5. When the asset later becomes `VALID`, aggregation may not be re-triggered — `validate_asset_metadata` only bumps the draft version's `modified` timestamp (not its status), while `validate_version_metadata` has already moved the version out of `PENDING` (`VALIDATING` → `VALID`), so later scheduled cycles would skip re-dispatching aggregation.

If that's what's happening, the summary would be frozen at the value computed before the replacement asset finished validating. This would also be consistent with the original 18-subject upload being correct while a single-file replacement is not.

## Possible directions (to confirm after tracing)

- Re-trigger aggregation when an asset finishes validating — e.g. in `validate_asset_metadata`, put the owning draft version(s) back into `Status.PENDING` on the success path so the next scheduled sweep re-runs aggregation. (Would need to confirm this doesn't cause a re-validation loop; `validate_pending_asset_metadata` appears to key off `Asset.status == PENDING`, not version status.)
- Or dispatch `aggregate_assets_summary_task` for the affected draft versions directly from the asset-validation completion path, which would also avoid transient undercounts.

Possibly related: #2091 ("Finalizing a zarr should put dandiset in `PENDING` state"), which looks like the same class of "downstream completion doesn't re-trigger the `PENDING` → aggregate cycle" issue.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing version_aggregate_assets_summary in dandiapi/api/services/metadata/__init__.py alongside validate_draft_version_metadata and the validation tasks in dandiapi/api/tasks/scheduled.py. Then inspect asset replacement and validation in dandiapi/api/services/asset/__init__.py; done means confirming the summary is recomputed after a replacement asset becomes VALID without creating a validation loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.