OHDSI / OHDSI/StudyAgent

Validate ACP cohort-method recommendations before script generation

Open
#41 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement good first issue
Dominant language
Python
Stars
12
Forks
7
Avg merge
2m
Merged PRs (30d)
12

Description

Problem

The cohort-method shell accepts ACP recommendations from the free-text analytic-parameters step and converts them into effective_analytic_settings, but there is no dedicated validation layer between the ACP response and downstream script generation.

This means malformed, incomplete, or semantically inconsistent ACP output can still flow into:

  • analysis-settings/cm_analytic_settings_recommendation.json
  • effective_analytic_settings
  • generated 06_cm_spec.R
  • generated analysisSpecification.json

Current behavior

  • ACP recommendations are converted through shell_settings_from_acp_recommendation(...)
  • If ACP returns source = acp_flow and status = ok, the mapped settings are used downstream
  • There is no explicit schema/semantic validation pass that confirms:
    • required settings are present
    • values are in supported ranges
    • combinations are coherent
    • unsupported fields are surfaced clearly

Risk

  • invalid generated scripts
  • silent fallback to partial defaults
  • hard-to-debug user experience when ACP output is plausible-looking but not executable

Requested change

Add a validation layer for cohort-method analytic recommendations before they are committed to shell state or used for script generation.

Acceptance criteria

  • ACP recommendation objects are validated before conversion/use
  • validation distinguishes:
    • schema errors
    • unsupported fields
    • semantically inconsistent settings
  • the shell surfaces clear actionable messages when validation fails
  • invalid recommendations do not silently produce executable-looking output
  • tests cover valid, partially valid, and invalid ACP recommendation payloads

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at shell_settings_from_acp_recommendation(...) and trace how ACP output reaches effective_analytic_settings and the generated artifacts. Compare valid, partially valid, and invalid recommendation payloads against the required schema, supported ranges, field support, and coherent combinations. Done means failures are clearly surfaced and invalid output does not produce executable-looking 06_cm_spec.R or analysisSpecification.json.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, r
Domain
backend, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.