E3SM-Project / E3SM-Project/simboard

[AI]: Add curated documentation retrieval for AI summary citations

Open
#188 1 comment 0 reactions 1 assignee Claimed by @tomvothecoder View on GitHub
type: ai
Dominant language
Python
Stars
1
Forks
2
Avg merge
24m
Merged PRs (30d)
15

Description

## Objective

SimBoard AI summaries need stronger source grounding for explanatory context, caveats, and citations.

Phase 3 should add a small curated retrieval path, starting with the v3.LR.historical ensemble Confluence page:

https://e3sm.atlassian.net/wiki/spaces/CM/pages/4182474754/v3.LR.historical+ensemble

This is not a broad RAG platform, AI agent, MCP integration, or source-code indexing effort.

## Scope

Add backend retrieval for curated documentation context in the existing AI summary workflow.

Initial source:

- v3.LR.historical ensemble Confluence export
- Use for: ensemble purpose, science context, caveats, terminology, interpretation notes
- Do not use for: simulation metadata, run status, provenance, execution IDs, artifact paths, or diagnostics availability

SimBoard metadata remains the source of truth for simulation-specific facts.

## Implementation notes

- Export the Confluence page to stable Markdown.
- Store source metadata: ID, title, original URL, export date, owner, scope, source type, authority level.
- Add a small retrieval abstraction rather than hardcoded prompt injection.
- Pass normalized retrieved context into the AI summary service.
- Return citation metadata with AI summary responses.
- Log retrieved source IDs and trace IDs.

Suggested interface:

`retrieve_context(query, simulation_metadata, limit) -> list[RetrievedContext]`

Each result should include source ID, title, URL/path, heading, excerpt, rank if available, and export/version metadata.

## Acceptance criteria

- AI summaries can retrieve context from the curated Confluence export.
- Documentation-grounded claims include citations.
- Simulation-specific claims still come from SimBoard metadata.
- Missing or unsupported context is shown as a caveat or limitation.
- Retrieval failure does not break baseline summary generation.
- Retrieval is behind a backend abstraction that can later connect to a larger RAG or knowledge service.
- Tests cover retrieval success, empty results, citation formatting, fallback behavior, and unsupported-claim avoidance.

## Out of scope

- Full RAG system
- AI agent workflow
- MCP integration
- Full Confluence sync
- Source-code indexing
- Open-ended doc chat
- Web retrieval
- User-uploaded document ingestion
- Production vector database

## Related

- Phase 3 of SimBoard AI assistance prototype
- Related issue: #52

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.