CorrelAid / CorrelAid/formtransform

[survey2ddi 2/6] Port normalize_responses: the LimeSurvey response-data path

Open
#11 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Step 2 of 6 for the survey2ddi retirement. Plan: HANDOVER_SURVEY2DDI.md. Independent of the xlsform2ddi --data issue; both are needed before survey2ddi's CLIs can be deprecated.

The gap

lstsvToDdiXml emits metadata. buildDataCsv (#8) emits a data CSV from submissions keyed by bare question name or group/name path. Neither shape is what a LimeSurvey response export produces, so there is no LimeSurvey full-mode path in this library.

limesurvey2ddi/transform.py in CorrelAid/survey2ddi does the missing step — normalize_responses(variables, rows):

  • Bracket subkeys. LimeSurvey exports a select_multiple as one column per subquestion, keyed with a bracket suffix. _match_choice(subkey, choices) maps that suffix back to the parsed choice code so it can line up with the <name>_<choice> DDI variable.
  • The underscore quirk. _norm(name) strips underscores and lowercases, "mirrors LimeSurvey's export behaviour" — LimeSurvey mangles field names on export, so matching must be done on the normalized form.

Both are LimeSurvey-specific and belong nowhere near the Kobo path.

What to build

src/pipelines/lstsv2ddi/data.ts, mirroring how xlsform2ddi/data.ts sits beside variables.ts:

lstsvToDataCsv(tsv: string, responses: Record<string, unknown>[], options?): string

Simplest honest shape: a key-normalizing adapter that re-keys LimeSurvey rows onto DDI variable names, then delegates to the existing buildDataCsv — the RFC 4180 writing and the column plan are already correct and must not be duplicated. Do not widen readCell in xlsform2ddi/data.ts with LimeSurvey heuristics; the two platforms' quirks stay separate.

Then add --data / --data-out to the lstsv2ddi CLI command, same flags and semantics as the xlsform2ddi issue.

Verification

  • Fixture-based unit test: a LimeSurvey response export (bracketed multi columns, underscore-mangled names) → correct 0/1 binaries and correct scalar columns
  • Header order equals <var name=""> order of lstsvToDdiXml on the same TSV
  • Byte-compare against Python normalize_responses + build_data_csv on one fixture, accounting for the known column-order divergence in the Python emitter (documented in the handover doc)
  • tests/fixtures/lstsv/ in survey2ddi has three usable structure TSVs

Acceptance criteria

  • Bracket-subkey → choice-code matching ported, with the _norm normalization
  • lstsv2ddi --data writes a CSV whose header matches the XML
  • LimeSurvey-specific keying lives in lstsv2ddi/, not in the shared emitter
  • Kobo path behaviour unchanged (existing data.test.ts still green)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with HANDOVER_SURVEY2DDI.md, src/pipelines/lstsv2ddi/data.ts, the existing xlsform2ddi/data.ts and variables.ts, and survey2ddi's limesurvey2ddi/transform.py. Compare the fixture-based output with Python normalize_responses plus build_data_csv, then run the existing data tests and lstsv2ddi CLI checks. Done means bracketed and underscore-mangled LimeSurvey fields produce the expected CSV, with headers matching the XML and Kobo tests unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
cli, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.