Viz.ChatGrid: add Context Management Module for Large SQL-Backed AI Chat Responses

Issue type

Feature Request

Issue applies to:

• PFLOW
• OPFLOW
• TCOPFLOW
• SCOPFLOW
• SOPFLOW
• CMake and build system
• Spack
• Visualization
• Documentation
• Other

Summary

Viz.ChatGrid currently passes raw SQL query results directly into the LLM prompt when generating chat responses.
When result sets are large (multi-state, multi-year, etc.), this causes:

• LLM context window overflow (current limit is 4k)
• Truncation of SQL output
• Biased answers caused by SQL ordering (e.g. only the first state like Alabama appears in a multi-state query)

This makes chat responses incorrect, misleading, or incomplete.
The system currently relies on auto-generated SQL filtering and ordering to control what fits into the LLM prompt, which is fragile and unsupervised.

CC @Walter462

Description how to reproduce the issue

There is no explicit layer responsible for:

• Token budgeting
• Result sampling / ranking
• Summarization
• Format-specific context shaping

The LLM receives raw database output, not curated context.

ExaGO version

develop at https://github.com/ORNL/ExaGO/commit/6c082a037f139432ed248f2823b61d8f90fe3411

System and environment details

All supported

Additional information

No response

Proposed solution

Add a Context Management Module between the database and ChatResponsePrep.

flowchart TB
  user>"USER"]
  user --"natural lang query"--> chat
subgraph "Front__End"
  chat[["CHAT"]]
  end
subgraph "BackEnd"
  LangChain[\"LangChain"/]
  DB[("DB")]
  LangChain --"srl request"-->DB
  ChatResponsePrep[\"Generate Natural language AI response for chat"/]
  OpenAIChatResponse["OpenAI"]
  chat --"natural lang query"--> LangChain
  ChatResponsePrep --"context enriched OpenAI request"--> OpenAIChatResponse
  OpenAIChatResponse --"sql-return context-powered reply"--> ChatResponsePrep
  ContextManagementModule(("Context
  Management
  Module"))
  DB --"raw data"--> ContextManagementModule
  ContextManagementModule --> ChatResponsePrep
  LangChain --"natural lang query"--> ChatResponsePrep 
  ChatResponsePrep --"human-readable reply"-----> chat
end
style ContextManagementModule fill:#000,stroke:#f55,stroke-width:4px,color:#fff,stroke-dasharray: 5 5

This module should:

• Enforce token budgets
• Select representative rows when full data doesn’t fit
• Perform summarization or aggregation where appropriate
• Provide structured, format-aware context to the LLM

Goals / Acceptance Criteria

The system should guarantee that:

• Multi-group queries (e.g., multi-state, multi-year) always include all groups
• No single category dominates context due to SQL ordering
• The LLM never receives raw, unbounded SQL dumps
• Context fits within LLM token limits while preserving correctness

Possible Implementations (non-binding)

• MCP-based context governance (schema, budgets, coverage guarantees)
• Result ranking + sampling
• Group-aware row selection
• SQL-native aggregation (GROUP BY, COUNT, SUM, AVG, etc.)
• Pre-LLM aggregation (per state, per year, etc.)
• RAG-style retrieval
• LLM-based summarization before final prompt