SACGF / SACGF/variantgrid

Classification upload API: run condition text automatch async, batched after accepting the upload

Open
#1,780 1 comment 0 reactions 1 assignee Assigned to @TheMadBug View on GitHub
Dominant language
Python
Stars
30
Forks
3
Avg merge
9h 28m
Merged PRs (30d)
42

Description

🤖 Written by Claude

## Problem

During a Shariant upload, the classification record API got very slow, and the server reported Monarch API timeouts. The cause is that condition text automatching runs **synchronously inside the upload request**, and can hit the external Monarch search API per record:

1. The sync upload posts records to `ClassificationView` (`classification/views/classification_view.py`), which runs `BulkClassificationInserter` in the web request.
2. Publishing a record fires `classification_post_publish_signal` (`classification/models/classification.py`) — a plain Django signal, so receivers run in-request.
3. The receiver `published()` in `classification/models/condition_text_matching.py` calls `sync_condition_text_classification(..., attempt_automatch=True)` → `top_level_suggestion()`.
4. If the condition text isn't an embedded ontology ID and doesn't match a local MONDO term, `search_suggestion()` calls `condition_text_search()` → HTTPS GET to `https://api.monarchinitiative.org/v3/api/search` (`classification/models/condition_text_search.py`).

Impact is amplified by:

- 60 second timeout (`MINUTE_SECS`) plus one retry with backoff on 5xx — up to ~2 minutes per record when Monarch is down.
- No caching on `top_level_suggestion` (the `@timed_cache` decorator is commented out), so repeated condition texts within a batch each re-hit the API.
- Failures are swallowed (`report_exc_info()`), so the upload doesn't fail — it just crawls.

## Proposed change

The automatch should be an **async job**, ideally run against the **whole batch** after the API request has been accepted and returned as quickly as possible:

- The upload request should only do the cheap, local work (create/sync `ConditionTextMatch` rows, update counts).
- After the batch is accepted, queue a Celery task that performs automatching for the affected `ConditionText`s — deduplicated across the batch, so each distinct condition text hits Monarch at most once.
- External Monarch failures then only affect the background job, never upload latency.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.