aws-samples / aws-samples/sample-kiro-cost-analyzer
Extract an ETL source-adapter seam (internal refactor, no behavior change)
- Dominant language
- Python
- Stars
- 11
- Forks
- 3
- Avg merge
- 3h 11m
- Merged PRs (30d)
- 36
Description
## Problem
The ETL pipeline is coupled to the two Kiro file formats at several well-localized but uncoordinated points: `ListFiles` decides `fileType: "csv" | "prompt"` from the S3 key; `parse_handler.py` branches on that string; `path_resolver.py`, `csv_schema_validator.py`, `normalizer.py` and `prompt_parser.py` each encode Kiro-specific knowledge. Adding a third source today means touching all of these and growing the `if file_type ==` ladder.
## Scope
Extract a **source-adapter** seam as a **purely internal refactor**: after it ships the pipeline behaves byte-for-byte as before, but a new source becomes a new adapter module + a registry entry, not a cross-cutting edit. This is the direct prerequisite for #56 (Bedrock ingestion).
## Design highlights
- `etl/sources/`: `base.py` (`SourceAdapter` protocol with `claim`/`parse`/`normalize` + `PathMetadata`), `kiro_csv.py`, `kiro_prompt_log.py`, `__init__.py` (ordered REGISTRY). Old modules are *moved behind* adapters, not rewritten.
- ListFiles iterates the registry; first `claim` wins; unclaimed keys skipped+logged as today.
- Deploy-window compat: `fileType` aliases `csv`→`kiro_csv`, `prompt`→`kiro_prompt_log` for in-flight Distributed Map items.
- Polymorphic cost fields: credit fields become optional-absent; add optional `inputTokens`/`outputTokens`/`cacheReadTokens`/`cacheWriteTokens`/`estimatedCostUsd`. Writer `ADD`s them only when present, so Kiro items keep their exact current attribute set.
- **Equivalence guaranteed by golden-model Hypothesis tests**: `adapter.normalize(adapter.parse(x)) == golden(x)` for both adapters, over generated valid inputs.
## Tasks
- [ ] `etl/sources/base.py` — protocol + `PathMetadata`
- [ ] `tests/test_source_adapter_equivalence.py` — golden copies + Hypothesis generators (written first)
- [ ] `kiro_csv.py` and `kiro_prompt_log.py` adapters
- [ ] Registry in `__init__.py` with deterministic order + aliases
- [ ] Point equivalence tests at adapters; both equal golden output
- [ ] Wire `list_handler.py` and `parse_handler.py` through the registry; delete the `if file_type ==` ladder
- [ ] Make credit fields optional-absent; add token/cost fields; extend `analytics_writer` ADD
- [ ] Dashboard-aggregation test tolerating items with no credit attributes
- [ ] Delete golden copies at close; update `docs/architecture.md`; `docs/changelog.md` entry
Full spec: `.kiro/specs/etl-source-adapter/` on the analysis branch.
Contributor guide
Research direction
Start with tests/test_source_adapter_equivalence.py and the full specification in .kiro/specs/etl-source-adapter/. Then trace etl/sources/base.py, list_handler.py, parse_handler.py, and analytics_writer to understand the registry, compatibility aliases, and optional fields. Done means both adapters match the golden outputs, existing Kiro attributes remain unchanged, and the listed tests and documentation updates are complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100