CorrelAid / CorrelAid/formtransform
[survey2ddi 2/6] Port normalize_responses: the LimeSurvey response-data path
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Step 2 of 6 for the survey2ddi retirement. Plan: HANDOVER_SURVEY2DDI.md. Independent of the xlsform2ddi --data issue; both are needed before survey2ddi's CLIs can be deprecated.
The gap
lstsvToDdiXml emits metadata. buildDataCsv (#8) emits a data CSV from submissions keyed by bare question name or group/name path. Neither shape is what a LimeSurvey response export produces, so there is no LimeSurvey full-mode path in this library.
limesurvey2ddi/transform.py in CorrelAid/survey2ddi does the missing step — normalize_responses(variables, rows):
- Bracket subkeys. LimeSurvey exports a
select_multipleas one column per subquestion, keyed with a bracket suffix._match_choice(subkey, choices)maps that suffix back to the parsed choice code so it can line up with the<name>_<choice>DDI variable. - The underscore quirk.
_norm(name)strips underscores and lowercases, "mirrors LimeSurvey's export behaviour" — LimeSurvey mangles field names on export, so matching must be done on the normalized form.
Both are LimeSurvey-specific and belong nowhere near the Kobo path.
What to build
src/pipelines/lstsv2ddi/data.ts, mirroring how xlsform2ddi/data.ts sits beside variables.ts:
lstsvToDataCsv(tsv: string, responses: Record<string, unknown>[], options?): string
Simplest honest shape: a key-normalizing adapter that re-keys LimeSurvey rows onto DDI variable names, then delegates to the existing buildDataCsv — the RFC 4180 writing and the column plan are already correct and must not be duplicated. Do not widen readCell in xlsform2ddi/data.ts with LimeSurvey heuristics; the two platforms' quirks stay separate.
Then add --data / --data-out to the lstsv2ddi CLI command, same flags and semantics as the xlsform2ddi issue.
Verification
- Fixture-based unit test: a LimeSurvey response export (bracketed multi columns, underscore-mangled names) → correct
0/1binaries and correct scalar columns - Header order equals
<var name="">order oflstsvToDdiXmlon the same TSV - Byte-compare against Python
normalize_responses+build_data_csvon one fixture, accounting for the known column-order divergence in the Python emitter (documented in the handover doc) tests/fixtures/lstsv/in survey2ddi has three usable structure TSVs
Acceptance criteria
- Bracket-subkey → choice-code matching ported, with the
_normnormalization -
lstsv2ddi --datawrites a CSV whose header matches the XML - LimeSurvey-specific keying lives in
lstsv2ddi/, not in the shared emitter - Kobo path behaviour unchanged (existing
data.test.tsstill green)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with HANDOVER_SURVEY2DDI.md, src/pipelines/lstsv2ddi/data.ts, the existing xlsform2ddi/data.ts and variables.ts, and survey2ddi's limesurvey2ddi/transform.py. Compare the fixture-based output with Python normalize_responses plus build_data_csv, then run the existing data tests and lstsv2ddi CLI checks. Done means bracketed and underscore-mangled LimeSurvey fields produce the expected CSV, with headers matching the XML and Kobo tests unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, typescript
- Domain
- cli, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100