EPIC: PSI Agent

Summary

Design and develop a PSI (Physical Science Informatics ) chat research/summarizer agent using the complete CARE process (Stages 1–5) to enable users to search, extract, analyze, and compare scientific data, documents, and publications, improving efficiency in large-scale scientific analysis and reducing manual effort in literature synthesis.

Motivation (WHY)

• Current PSI workflows require users to:
  • manually search across datasets, reports, and publications
  • extract variables and numerical data manually
  • compare findings across multiple studies
• Handling diverse formats (tables, text, images) is complex and time-consuming
• No intelligent system exists to automate cross-study synthesis and analysis

Acceptance Criteria (DONE =)

• Full implementation of CARE Stages 1–5
• Agent supports:
  • search across PSI datasets, documents, and publications
  • extraction of variables, parameters, and numerical values
  • cross-study comparison and trend identification
• MCP Tool integration validated (if available)
• Synthetic benchmark dataset created and executed
• Gold benchmark dataset validated with domain experts
• Measurable evaluation metrics:
  • relevance of retrieved information
  • accuracy of extracted data
  • quality of summaries and comparisons

Sub-Issues / Execution Plan

• CARE Design: Conduct CARE Stages 1–4 design sessions to define PSI agent architecture, data sources, and scientific workflows.
• System Prompt: Develop, refine, and finalize the system prompt to support scientific analysis, comparison, and extraction tasks.
• PSI Data Integration: Identify and integrate PSI data sources (datasets, publications, reports, tables) into the retrieval pipeline.
• MCP Tool Integration: Collaborate with Dev support of PSI to clarify availability, design integration approach, and validate MCP Tool for PSI workflows.
• Benchmarking Strategy: Define benchmarking framework, datasets, and success metrics (accuracy, relevance, analytical quality).
• Synthetic Testing: Execute synthetic benchmark evaluations and compare performance against baseline approaches.
• Gold Standard Testing: Develop and execute gold-standard benchmarks with SMEs and validate scientific accuracy.
• Documentation: Document results, analyze performance gaps, and provide recommendations for improvement.
• Deployment & Validation: Deploy agent within AKD ecosystem and validate with scientists and domain experts.