aws-samples / aws-samples/sample-kiro-cost-analyzer

Extract an ETL source-adapter seam (internal refactor, no behavior change)

Closed
#55 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
11
Forks
3
Avg merge
3h 11m
Merged PRs (30d)
36

Description

## Problem

The ETL pipeline is coupled to the two Kiro file formats at several well-localized but uncoordinated points: `ListFiles` decides `fileType: "csv" | "prompt"` from the S3 key; `parse_handler.py` branches on that string; `path_resolver.py`, `csv_schema_validator.py`, `normalizer.py` and `prompt_parser.py` each encode Kiro-specific knowledge. Adding a third source today means touching all of these and growing the `if file_type ==` ladder.

## Scope

Extract a **source-adapter** seam as a **purely internal refactor**: after it ships the pipeline behaves byte-for-byte as before, but a new source becomes a new adapter module + a registry entry, not a cross-cutting edit. This is the direct prerequisite for #56 (Bedrock ingestion).

## Design highlights

- `etl/sources/`: `base.py` (`SourceAdapter` protocol with `claim`/`parse`/`normalize` + `PathMetadata`), `kiro_csv.py`, `kiro_prompt_log.py`, `__init__.py` (ordered REGISTRY). Old modules are *moved behind* adapters, not rewritten.
- ListFiles iterates the registry; first `claim` wins; unclaimed keys skipped+logged as today.
- Deploy-window compat: `fileType` aliases `csv`→`kiro_csv`, `prompt`→`kiro_prompt_log` for in-flight Distributed Map items.
- Polymorphic cost fields: credit fields become optional-absent; add optional `inputTokens`/`outputTokens`/`cacheReadTokens`/`cacheWriteTokens`/`estimatedCostUsd`. Writer `ADD`s them only when present, so Kiro items keep their exact current attribute set.
- **Equivalence guaranteed by golden-model Hypothesis tests**: `adapter.normalize(adapter.parse(x)) == golden(x)` for both adapters, over generated valid inputs.

## Tasks

- [ ] `etl/sources/base.py` — protocol + `PathMetadata`
- [ ] `tests/test_source_adapter_equivalence.py` — golden copies + Hypothesis generators (written first)
- [ ] `kiro_csv.py` and `kiro_prompt_log.py` adapters
- [ ] Registry in `__init__.py` with deterministic order + aliases
- [ ] Point equivalence tests at adapters; both equal golden output
- [ ] Wire `list_handler.py` and `parse_handler.py` through the registry; delete the `if file_type ==` ladder
- [ ] Make credit fields optional-absent; add token/cost fields; extend `analytics_writer` ADD
- [ ] Dashboard-aggregation test tolerating items with no credit attributes
- [ ] Delete golden copies at close; update `docs/architecture.md`; `docs/changelog.md` entry

Full spec: `.kiro/specs/etl-source-adapter/` on the analysis branch.

Contributor guide

Open the contributing guide

Research direction

Start with tests/test_source_adapter_equivalence.py and the full specification in .kiro/specs/etl-source-adapter/. Then trace etl/sources/base.py, list_handler.py, parse_handler.py, and analytics_writer to understand the registry, compatibility aliases, and optional fields. Done means both adapters match the golden outputs, existing Kiro attributes remain unchanged, and the listed tests and documentation updates are complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.