databrickslabs / databrickslabs/ontos

[Improvement]: Term Mapping bulk-accept should confirm/guard against low-confidence rows

Open
#482 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

roadmap/v1.2.0 scope/ontology status/needs-triage tech/javascript type/improvement
Dominant language
Python
Stars
212
Forks
71
Avg merge
4d 10h
Merged PRs (30d)
43

Description

Context

Surfaced while smoke-testing the Term Mapping workbench (PR #481, PRD #469). The bulk Accept N action in suggestion-queue-table.tsx accepts every selected row indiscriminately, regardless of confidence bucket. Combined with the DataTable's page-scoped "Select all" checkbox, the natural "select all → accept" gesture happily promotes 50% noise hits like `transaction_id → productId` or `store_id → orderId` into the accepted bucket where a single click on Apply can persist them as real semantic links.

The suggester is doing its job — it scores those low (50–70%) and surfaces a "Why" explanation — but the workbench gives no friction before turning that into a write.

Repro:

  1. Run `/term-mapping`, upload a small customer ontology, create a run with default filters.
  2. In the suggestion queue, click the header Select all checkbox (selects 10 rows on the first page).
  3. Click Accept 10.
  4. Observe: all 10 are accepted, including obvious mismatches with confidence in the 50–60% range.

Proposed UX

Tiered guard-rail, in order of effort:

  • Minimum: when the selection contains rows below a configurable threshold (default 0.7), show a confirm dialog summarising the confidence distribution before submitting decisions, e.g. "Accept 10 selected: 1 high (≥80%), 2 medium (60–80%), 7 low (<60%). Continue?". Reuse the existing apply-dialog visual treatment.
  • Better: default the suggestion table sort to confidence DESC and surface filter chips for High / Medium / Low so the obvious move becomes "accept all High".
  • Best: add a one-click "Accept all ≥ X%" quick action above the table that operates on the full pending set, not just the current page (today's Select all is page-scoped, which is its own footgun).

Acceptance criteria

  • Bulk-accepting a selection that contains any suggestion below the configured threshold prompts a confirmation summarising the per-bucket counts.
  • User can dismiss the dialog (no decisions persisted) or proceed (existing POST behaviour).
  • Threshold is sourced from the same scoring constants as the suggester (AUTO_ACCEPT / a new LOW_CONFIDENCE cutoff in controller/term_mapping/scoring.py) so backend and UI stay in lockstep.
  • Page-scoped vs. full-pending "Select all" behaviour is either documented in a tooltip or replaced with a "Select all 18 pending" affordance.

Related

  • PRD: #469
  • Backend PR: #480
  • UI PR: #481 (current implementation)
  • Phase 4 (LLMJudgeSuggester re-ranker) will reduce the number of low-confidence noise candidates that ever reach the queue, but is complementary to this guard-rail rather than a replacement.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in suggestion-queue-table.tsx and trace the bulk Accept action and page-scoped selection behavior. Read controller/term_mapping/scoring.py for AUTO_ACCEPT or LOW_CONFIDENCE, then inspect the existing apply-dialog visual treatment. Done means low-confidence bulk selections prompt with per-bucket counts, dismissal persists nothing, proceeding keeps the existing POST behavior, and selection scope is documented or improved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
backend, frontend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.