Validate ACP cohort-method recommendations before script generation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 12
- Forks
- 7
- Avg merge
- 2m
- Merged PRs (30d)
- 12
Description
Problem
The cohort-method shell accepts ACP recommendations from the free-text analytic-parameters step and converts them into effective_analytic_settings, but there is no dedicated validation layer between the ACP response and downstream script generation.
This means malformed, incomplete, or semantically inconsistent ACP output can still flow into:
analysis-settings/cm_analytic_settings_recommendation.jsoneffective_analytic_settings- generated
06_cm_spec.R - generated
analysisSpecification.json
Current behavior
- ACP recommendations are converted through
shell_settings_from_acp_recommendation(...) - If ACP returns
source = acp_flowandstatus = ok, the mapped settings are used downstream - There is no explicit schema/semantic validation pass that confirms:
- required settings are present
- values are in supported ranges
- combinations are coherent
- unsupported fields are surfaced clearly
Risk
- invalid generated scripts
- silent fallback to partial defaults
- hard-to-debug user experience when ACP output is plausible-looking but not executable
Requested change
Add a validation layer for cohort-method analytic recommendations before they are committed to shell state or used for script generation.
Acceptance criteria
- ACP recommendation objects are validated before conversion/use
- validation distinguishes:
- schema errors
- unsupported fields
- semantically inconsistent settings
- the shell surfaces clear actionable messages when validation fails
- invalid recommendations do not silently produce executable-looking output
- tests cover valid, partially valid, and invalid ACP recommendation payloads
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at shell_settings_from_acp_recommendation(...) and trace how ACP output reaches effective_analytic_settings and the generated artifacts. Compare valid, partially valid, and invalid recommendation payloads against the required schema, supported ranges, field support, and coherent combinations. Done means failures are clearly surfaced and invalid output does not produce executable-looking 06_cm_spec.R or analysisSpecification.json.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, r
- Domain
- backend, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100