[PRD]: Complete Search & Discovery — Faceted Filtering, Rich Results, and Discovery UX

Problem Statement

Ontos has a functional global search (Index Search) and an AI-powered conversational search (Ask Ontos), but the discovery experience is incomplete. The Index Search returns a flat, unfiltered list of results with no way to narrow by type, domain, status, owner, or certification level. Search results show only a title and truncated description — no metadata badges, no tags, no status indicators. There is no structured query syntax beyond tag:, no sort options, no pagination, and no result counts. The header search bar and full search page have inconsistent UX. The LLM's global_search tool uses a separate scoring algorithm from SearchManager, causing result divergence. Consumers have no browse/discovery surface on their Home Page. As the number of indexed assets grows, the current experience will not scale.

Solution

Complete the Search & Discovery feature with faceted filtering, rich result cards, extended query syntax, consistent ranking across all search surfaces, and discovery sections on the consumer Home Page. The SearchManager becomes the single deep module handling all matching, filtering, ranking, and pagination — consumed by the API, the header search bar, the full search page, and the LLM tools.

Key capabilities added:

• Faceted filter sidebar on the Index Search page with checkboxes/pills for Type, Domain, Status, Owner, and (future) Certification Level
• Rich result cards showing type badge, status badge, domain, tags, certification level, and description
• Extended query syntax: type:data-product status:active domain:finance owner:jane parsed by SearchManager alongside free-text
• Sort options: Relevance (default), Newest, Alphabetical
• Result count summary and facet counts (e.g., "3 Data Products, 5 Contracts, 12 Assets")
• Pagination via limit/offset on the API with "Load more" in the UI
• Hierarchical domain filtering: selecting a parent domain includes children
• "My Items" quick filter: pre-filters by owner == current_user
• Certification/recency ranking boosts: Gold-certified and recently updated items rank higher
• Header search bar improvements: compact suggestions (title + type), "Search for '{query}' →" navigation link
• global_search tool convergence: delegates to SearchManager instead of reimplementing scoring
• Home Page discovery sections for consumer persona: "Recently Published," "Highest Certified," "Your Subscriptions"
• Search config cleanup: remove orphaned glossary-term config, add certification boost and recency settings
• Recent search history: localStorage-based, shown when search box is focused
• Contextual "search within": entity pages (e.g., Domain detail) offer pre-filtered search links
• Search analytics logging: queries + result counts logged for zero-result analysis

User Stories

1. As a data consumer, I want to filter search results by asset type (Data Product, Data Contract, Asset, Domain, etc.), so that I can focus on the category I care about.
2. As a data consumer, I want to filter search results by domain, so that I can discover assets within my business area.
3. As a data consumer, I want domain filtering to be hierarchical, so that selecting "Finance" also shows items in sub-domains like "Finance/Risk."
4. As a data consumer, I want to filter search results by status (Active, Draft, Deprecated, etc.), so that I can find only production-ready assets.
5. As a data producer, I want a "My Items" quick filter, so that I can quickly find assets I own.
6. As a data producer, I want to filter search results by owner, so that I can find assets owned by a specific team member.
7. As a data consumer, I want each search result to show type, status, domain, top tags, and certification level as inline badges, so that I can evaluate results without clicking into each one.
8. As a data consumer, I want to see a result count summary at the top of search results (e.g., "3 Data Products, 5 Contracts"), so that I know the scope of matches.
9. As a user, I want to sort search results by Relevance, Newest, or Alphabetical, so that I can find what I need faster.
10. As a power user, I want to type structured filters in the search box (e.g., type:data-product status:active pipeline), so that I can search quickly without using a filter sidebar.
11. As a user, I want search results to be paginated (or "Load more"), so that the page doesn't lag when there are hundreds of results.
12. As a data consumer, I want Gold-certified and recently updated assets to rank higher in search results, so that the most trusted and current items appear first.
13. As a user, I want the header search bar to show compact suggestions (title + type) and a "View all results →" link, so that I can quickly navigate or jump to the full search page.
14. As a user, I want Ask Ontos and Index Search to return consistent results for the same query, so that I'm not confused by different rankings.
15. As a data consumer, I want my Home Page to show "Recently Published," "Highest Certified," and "Your Subscriptions" sections, so that I can discover data products without searching.
16. As a user, I want to see my recent searches when I focus the search box, so that I can quickly re-run previous queries.
17. As a user viewing a Domain detail page, I want a "Search within this domain" affordance, so that I can discover assets within that domain easily.
18. As an admin, I want to see which search queries return zero results, so that I can improve search configuration and data quality.
19. As a data steward, I want certification level to be a filterable dimension in search, so that I can find all uncertified or Bronze-level assets that need attention.
20. As a data consumer, I want to filter marketplace/published products by minimum certification level, so that I only see trusted data products.
21. As a user, I want the search config to support certification boost and recency decay settings, so that admins can tune ranking behavior.
22. As a developer integrating with Ontos via the LLM tool layer, I want the global_search tool to use the same ranking as the search API, so that results are consistent across interfaces.

Implementation Decisions

Deep Module: SearchManager

The SearchManager is the single deep module that encapsulates all search complexity behind a simple interface. Its public API expands from search(query, auth, user) to:

search(query, auth, user, filters?, sort?, limit?, offset?) -> SearchResult

Where SearchResult contains results: List[SearchIndexItem], total_count: int, and facets: Dict[str, Dict[str, int]].

Internally it:

• Parses structured query syntax (type:, status:, domain:, owner:) from the query string, separating filters from free-text
• Applies free-text matching using existing field configs (prefix, substring, exact, fuzzy)
• Applies structured filters as post-match predicates (or pre-filter for efficiency)
• Computes facet counts from the filtered result set
• Applies certification boost and recency decay to ranking scores
• Handles hierarchical domain filtering by walking the domain tree
• Paginates results
• Returns the structured SearchResult envelope

SearchIndexItem Expansion

Add created_at: Optional[datetime] and updated_at: Optional[datetime] fields. All searchable managers populate these from their source entities during get_search_index_items() and _notify_index_upsert().

Search Routes Expansion

GET /api/search gains query params:

• type (comma-separated, e.g., ?type=data-product,data-contract)
• status (comma-separated)
• domain_id (UUID, with hierarchical expansion)
• owner (string)
• sort (enum: relevance, newest, alphabetical)
• limit (int, default 50)
• offset (int, default 0)

Response changes from List[SearchIndexItem] to SearchResult envelope.

GlobalSearchTool Convergence

The global_search tool in src/tools/search.py stops reimplementing scoring and instead calls SearchManager.search() (without permission filtering, since the LLM operates in a trusted context). The tool's parameters map to SearchManager filters.

Frontend: SearchFilterSidebar Component

New component rendered alongside IndexSearch. Receives facet counts from the API response. Renders checkbox groups for Type, Status, Domain (hierarchical tree), and Owner. Includes a "My Items" toggle. Filter changes update URL query params and trigger a new search.

Frontend: SearchResultCard Component

Replaces the current <a> element in IndexSearch. Shows:

• Icon (from feature/type, existing logic)
• Title (clickable link)
• Description (truncated)
• Inline badges: type pill, status pill, domain tag, certification badge (when lifecycle work lands), top 2-3 tags

Frontend: Header Search Bar Enhancement

search-bar.tsx dropdown gets:

• Compact result format: icon + title + type label (no description)
• "Search for '{query}' in all results →" link at bottom, navigating to /search/index?query=...

Frontend: Home Page Discovery Sections

Consumer persona's Home Page gains:

• "Recently Published" — GET /api/search?status=active&sort=newest&limit=5 (or dedicated endpoint)
• "Highest Certified" — GET /api/search?sort=certification&limit=5 (requires lifecycle work)
• "Your Subscriptions" — existing subscription data

These sections render as horizontal card lists.

Search Config Changes

• Remove orphaned glossary-term from search_config.yaml
• Add ranking.certification_boost: float (default 1.5) — multiplier for certified items
• Add ranking.recency_decay: bool (default true) — boost recently updated items
• Add ranking.recency_half_life_days: int (default 90) — half-life for recency decay

Recent Search History

Client-side only, stored in localStorage. Array of last 10 queries. Shown as a dropdown when the search input is focused and empty. No backend involvement.

Contextual Search Links

Entity detail pages (Domain, Project, Team) get a small "Search within" input or link that navigates to /search/index?domain_id=... or /search/index?query=domain:....

Search Analytics Logging

Lightweight: log {query, result_count, user_id, timestamp} to structured backend logs (or a search_queries table if admin UI is desired). Zero-result queries flagged for admin review.

Testing Decisions

What makes a good test

Tests should verify external behavior through the public interface, not internal implementation details. A good search test provides input (query + filters + index state) and asserts on output (result set, ordering, counts). Tests should not assert on internal scoring calculations or intermediate data structures.

Modules to test

1. SearchManager (unit tests) — highest priority:

   • Structured query parsing (type:data-product status:active pipeline → filters + free-text)
   • Facet count computation
   • Pagination (limit/offset, total_count accuracy)
   • Certification boost (certified items rank higher than uncertified for equal text match)
   • Recency decay (recently updated items rank higher)
   • Hierarchical domain filtering (parent domain includes children)
   • "My Items" owner filtering
   • Sort options (relevance, newest, alphabetical)
   • Backward compatibility (existing tag: syntax still works)

2. Search Routes (integration tests):

   • New query params accepted and forwarded correctly
   • Response envelope structure (results, total_count, facets)
   • Permission filtering still works with new params
   • Invalid params return 400

3. GlobalSearchTool (unit test):

   • Delegates to SearchManager and returns consistent results

4. SearchFilterSidebar (component test, nice-to-have):

   • Renders facets from API response
   • Filter changes update URL params
   • "My Items" toggle works

5. SearchResultCard (component test, nice-to-have):

   • Renders all badge types correctly
   • Handles missing optional fields gracefully

Prior art

• src/backend/src/tests/unit/test_search_manager.py — existing unit tests for SearchManager
• src/backend/src/tests/unit/test_search_registry.py — registry tests
• src/backend/src/tests/integration/test_search_routes.py — integration tests for search routes

Out of Scope

• Dedicated Marketplace view — Phase 8 of the Unified Lifecycle Tracking plan (#94) already covers marketplace filters. This PRD enhances the general search; marketplace is a separate concern.
• Saved searches — Bookmarkable filter+query combos require a backend table and more UI. Defer until there are many users with complex filter combos.
• Full autocomplete/typeahead engine — True typeahead with prefix indexes is a larger effort. The header bar already does debounced search; we improve the presentation, not the engine.
• External catalog API — Token-authenticated API for notebooks/CI is a separate "Catalog API" feature, not search.
• Ontology ↔ Index cross-links — Linking KG concepts to index results is a separate UX concern.
• Elasticsearch / Vector Search migration — The in-memory index is fine for current scale. When it outgrows memory, migrate to a real search backend. This PRD keeps the in-memory approach.
• Related items / graph exploration — Relationship-based discovery belongs on detail pages and in Ask Ontos, not in index search results.

Further Notes

• Dependency on Lifecycle Tracking: Certification and publication badges in search results depend on the Unified Lifecycle Tracking work (#86, specifically Phases 3-4). The search infrastructure (filter dimensions, ranking boosts) should be built to accept these signals, but the actual data won't be available until those phases land. Use feature flags or graceful null-handling.
• Incremental delivery: The vertical slices should be ordered so that each one is independently demoable. The first slice (structured query parsing + API params) unblocks both the filter sidebar and the extended query syntax.
• Migration path: The response format change from List[SearchIndexItem] to SearchResult envelope is a breaking change for the frontend. Both index-search.tsx and search-bar.tsx must be updated in the same slice that changes the API response.

Start by reading the existing SearchManager and the GET /api/search entry point, then inspect src/tools/search.py and the frontend search-bar.tsx. Review the SearchManager unit-test, route integration-test, and GlobalSearchTool test requirements before choosing a focused slice. Done means the selected filters, ranking or result presentation work through the public interface with corresponding behavior tests.