Classification upload API: run condition text automatch async, batched after accepting the upload
- Dominant language
- Python
- Stars
- 30
- Forks
- 3
- Avg merge
- 9h 28m
- Merged PRs (30d)
- 42
Description
🤖 Written by Claude
## Problem
During a Shariant upload, the classification record API got very slow, and the server reported Monarch API timeouts. The cause is that condition text automatching runs **synchronously inside the upload request**, and can hit the external Monarch search API per record:
1. The sync upload posts records to `ClassificationView` (`classification/views/classification_view.py`), which runs `BulkClassificationInserter` in the web request.
2. Publishing a record fires `classification_post_publish_signal` (`classification/models/classification.py`) — a plain Django signal, so receivers run in-request.
3. The receiver `published()` in `classification/models/condition_text_matching.py` calls `sync_condition_text_classification(..., attempt_automatch=True)` → `top_level_suggestion()`.
4. If the condition text isn't an embedded ontology ID and doesn't match a local MONDO term, `search_suggestion()` calls `condition_text_search()` → HTTPS GET to `https://api.monarchinitiative.org/v3/api/search` (`classification/models/condition_text_search.py`).
Impact is amplified by:
- 60 second timeout (`MINUTE_SECS`) plus one retry with backoff on 5xx — up to ~2 minutes per record when Monarch is down.
- No caching on `top_level_suggestion` (the `@timed_cache` decorator is commented out), so repeated condition texts within a batch each re-hit the API.
- Failures are swallowed (`report_exc_info()`), so the upload doesn't fail — it just crawls.
## Proposed change
The automatch should be an **async job**, ideally run against the **whole batch** after the API request has been accepted and returned as quickly as possible:
- The upload request should only do the cheap, local work (create/sync `ConditionTextMatch` rows, update counts).
- After the batch is accepted, queue a Celery task that performs automatching for the affected `ConditionText`s — deduplicated across the batch, so each distinct condition text hits Monarch at most once.
- External Monarch failures then only affect the background job, never upload latency.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.