awslabs / awslabs/loom

Provide more granular cost reporting

Open
#21 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
183
Forks
41
Avg merge
8h 6m
Merged PRs (30d)
2

Description

## Overview

Loom's cost dashboard currently reports costs aggregated per agent, with a date-range filter and an existing (agent-tag-based) group filter. It does not report per-user cost attribution at all, and its "group" filter is really an agent-level `loom:group` tag rather than a first-class grouping dimension. Since session records already carry a `user_id` and agent tagging already exists, extend cost reporting to break down costs by user and by group, and give users flexibility to query/filter cost attribution data beyond the fixed agent/date-range view that exists today.

## Context

### Current State

- `GET /api/dashboard/costs` (`backend/app/routers/costs.py` L42-186) is the primary cost endpoint. It queries `Invocation` joined to `InvocationSession` (L73-75), filters by `agent_id.in_(agent_ids)` and an optional `days` window on `Invocation.created_at` (L77-79), and aggregates SUM of `input_tokens`, `output_tokens`, `estimated_cost`, `client_duration_ms`, `idle_cpu_cost`, `idle_memory_cost`, `stm_cost`, `ltm_cost` **per agent** (L110-117), recomputing CPU/memory runtime cost from duration using live pricing constants (L119-129, `AGENTCORE_RUNTIME_PRICING` in `backend/app/routers/agents.py`)
- A `group` query parameter exists (L56-65, L215-224) but it resolves to `Agent.get_tags()["loom:group"]` — it filters which *agents* are included based on an agent-level tag value, not a true independent cost-grouping dimension. There is no per-user equivalent at all
- `POST /api/dashboard/costs/actuals` (L194-426) pulls real CloudWatch usage (`get_usage_log_events_by_time`, `get_memory_log_events`) aggregated by `(agent_name, session_id)` — the only actual-billing-adjacent data path, computed on demand and not persisted
- `frontend/src/pages/CostDashboardPage.tsx` shows summary cards, an "Estimated Costs" table per agent, and an "Actual Costs" section with expandable per-agent → per-session rows. The only filter control is a date-range button group (7/30/90/All days, L208-221); there is no agent selector, no user filter, and no free-text/custom query UI
- `backend/app/models/session.py` (`InvocationSession`, L8-39) has a `user_id` column (L21, `String, nullable=True, index=True`) — user identity **is** already captured per session
- `backend/app/models/invocation.py` (`Invocation`, L8-101) has **no** `user_id` column — only `session_id`. Any per-user cost aggregation must join `Invocation` → `InvocationSession.user_id`
- There is no backend `User` model and no backend "group" table. User identity is JWT-derived only: `UserInfo` (`backend/app/dependencies/auth.py` L129-148: `sub`, `username`, `groups`, `scopes`, `idp_type`) built from the Cognito `cognito:groups` claim or an external IdP's configurable `group_claim_path`. "Group," for ABAC purposes, means Cognito/IdP group membership (`g-admins-*`/`g-users-*`, `GROUP_SCOPES` in `auth.py` L45-87 and mirrored in `frontend/src/contexts/AuthContext.tsx` L46); this is a **different concept** from the `loom:group` agent tag currently used to filter the cost dashboard, and the two should not be conflated when this issue says "per group"
- Tags attach only to resources (`Agent.tags`, `Memory.tags` — JSON blob columns with `get_tags()`/`set_tags()` helpers), not to `Invocation` or `InvocationSession` rows. There is no tag table joined to individual invocations/sessions, so any cost-center/team grouping today only exists via the agent's own `loom:group` tag, not per-invocation

### Key Files

- `backend/app/routers/costs.py` — existing cost dashboard/actuals endpoints
- `backend/app/models/invocation.py`, `backend/app/models/session.py` — the substrate any per-user/per-group aggregation must query; `InvocationSession.user_id` exists, `Invocation` has no user column
- `backend/app/dependencies/auth.py`, `frontend/src/contexts/AuthContext.tsx` — `UserInfo`/group-claim resolution and `GROUP_SCOPES`
- `backend/app/models/tag_policy.py`, `backend/app/models/tag_profile.py` — tag key/policy definitions (agent/memory scoped only)
- `frontend/src/pages/CostDashboardPage.tsx` — cost dashboard UI, currently date-range-only filtering
- `frontend/src/pages/RegistryPage.tsx` — precedent for free-text search + status/type dropdown filters (L368-372, `registryApi.searchRegistry()`)
- `frontend/src/pages/CatalogPage.tsx` — precedent for faceted tag-key/value filtering (`tagFilters` state L75, `AddFilterDropdown`/`MultiSelect` components, filter state persisted to `localStorage`)

## Requirements

### R1: Per-user and per-group cost metrics

Users must be able to view cost metrics broken down by individual user and by group, in addition to the existing per-agent breakdown.

- Add a new aggregation path in `backend/app/routers/costs.py` (or a new endpoint, e.g. `GET /api/dashboard/costs/by-user` and `GET /api/dashboard/costs/by-group`) that joins `Invocation` → `InvocationSession.user_id` and aggregates the same cost/token/duration fields currently summed per-agent (L110-117), grouped by `user_id` instead of (or in addition to) `agent_id`
- Define what "group" means for this requirement and make the choice explicit rather than implicit: either (a) reuse the existing `loom:group` agent-tag convention already wired into the `group` query param, accepting that it is agent-scoped rather than truly attributable per invocation, or (b) resolve group membership from the authenticated user's IdP/Cognito groups (`UserInfo.groups`) at query time, joining session `user_id` values to their group membership as of query time. Document the chosen semantics, including how historical invocations are attributed if a user's group membership changes after the invocation occurred
- Handle `user_id IS NULL` sessions (anonymous/service invocations, or historical rows predating this feature) as an explicit "Unattributed" bucket in both breakdowns, rather than silently dropping them from totals
- `CostDashboardPage.tsx` must add a per-user and per-group view (e.g. new tabs or a breakdown-dimension toggle alongside the existing per-agent table), reusing the existing table/expandable-row patterns already used for per-agent/per-session actuals
- Respect existing ABAC scopes (`costs:read`/`costs:write` per the sidebar/page gating established in the navigation-simplification work) — a user viewing per-user costs should not see cost data for users/groups outside what their own scope permits, if such a restriction is meaningful for this deployment (clarify with stakeholders whether cost data visibility should itself be scoped by group membership, e.g. a group admin sees only their group's users)

### R2: Flexible / custom querying of cost attribution data

Users must have some flexibility to run custom queries or searches over cost attribution data, rather than being limited to the fixed date-range-only filter that exists today.

- Add filter controls to the cost dashboard beyond date range: at minimum, filter by agent, by user, and by group (per R1), combinable (e.g. "show me costs for agent X, user Y, last 30 days")
- Follow one of the two existing UI precedents in this codebase rather than inventing a new pattern: either `RegistryPage.tsx`'s free-text search + dropdown-filter model, or `CatalogPage.tsx`'s faceted multi-select filter model (tag-key/value chips with `AddFilterDropdown`/`MultiSelect`, persisted to `localStorage`) — the faceted model is likely the better fit since cost attribution filtering is inherently multi-dimensional (agent × user × group × date range)
- Backend query endpoint(s) must accept combinable filter parameters (agent_id(s), user_id(s), group value(s), date range) and return aggregated results scoped to the intersection of all provided filters, mirroring the existing `agent_ids`/`group`/`days` parameter style in `costs.py` rather than introducing a wholly different query DSL
- Consider whether "custom query" should also support ad hoc grouping/sort (e.g. group-by user vs. group-by agent vs. group-by day) as a dimension selector, rather than requiring a separate endpoint per breakdown — this avoids proliferating near-duplicate endpoints as more attribution dimensions are added later
- Results must remain exportable/readable in the same table format already used elsewhere in the cost dashboard, so this doesn't introduce a visually inconsistent "reporting" sub-experience

## Additional Considerations

- **R3: `Invocation` lacks a `user_id` column** — Per-invocation cost rows only carry `session_id`; user attribution requires a join through `InvocationSession`. If per-invocation-level user filtering (not just per-session) is needed for R2's custom queries, consider whether denormalizing `user_id` onto `Invocation` (backfilled via join, populated going forward at invocation time) would simplify query performance versus always joining, especially as invocation volume grows.
- **R4: Group semantics need a decision, not an assumption** — This issue's "group" is ambiguous between the ABAC/Cognito group model and the `loom:group` agent tag. Get explicit sign-off on which one (or both, distinctly labeled) this feature reports on before implementation, since they have different data sources, different failure modes (tag missing vs. group-claim missing), and different security implications (agent tags are mutable metadata; Cognito groups are identity-provider-asserted).
- **R5: Historical data backfill** — Existing `InvocationSession` rows may already have `user_id` populated (need to confirm at implementation time) but any invocations predating group-tag adoption or predating this feature will have incomplete attribution. Decide how "Unattributed" buckets are surfaced (R1) and whether a backfill/reconciliation job is needed for older data.
- **R6: Cost-of-query risk** — Ad hoc, combinable filters over `Invocation`/`InvocationSession` (R2) could produce expensive unindexed queries at scale (e.g. filtering by `user_id` across all agents and all time). Ensure appropriate indexes exist (`InvocationSession.user_id` is already indexed per session.py L21; confirm equivalent coverage for any new join paths) and consider capping unbounded date ranges for custom queries.
- **R7: Privacy/visibility scoping** — Per-user cost reporting surfaces individual usage data. Consider whether users should be able to view only their own costs by default, with elevated scope required to view others' — this is a policy decision for stakeholders, not purely a technical one, and should be resolved before the UI ships an unrestricted "browse by any user" experience.
- **R8: Export/reporting format** — If custom queries are meant to support anything resembling ad hoc financial reporting (chargebacks, cost allocation to teams), consider whether CSV/JSON export of query results is needed alongside the in-app table view, since finance/ops stakeholders typically need to move this data into spreadsheets or BI tools rather than only viewing it in-app.

## Testing

- Run backend tests: `cd backend && make test`
- Run frontend typecheck: `cd frontend && npx tsc --noEmit`
- Verify per-user cost breakdown sums match the existing per-agent totals when aggregated back up (i.e. summing all users' costs for a given agent/date range equals the existing per-agent total) — a correctness check against double-counting or dropped rows
- Verify sessions/invocations with `user_id IS NULL` appear in an explicit "Unattributed" bucket and are not silently excluded from totals
- Verify per-group breakdown behaves correctly under whichever group semantics are chosen (R4), including a case where a user's current group membership differs from what it was historically, if that scenario is in scope
- Verify combinable filters (agent + user + group + date range) in the custom query UI return the correct intersection, including edge cases (no matching rows, all filters cleared, filters that reference a deleted agent/user)
- Verify ABAC scoping (R1's last bullet) restricts visibility appropriately if implemented
- Manually exercise the new cost dashboard views and filter UI in the browser, not just via typecheck/unit tests

## Out of Scope

- Building a general-purpose BI/reporting tool or query language — "custom queries" here means combinable structured filters over existing cost dimensions, not a SQL-like ad hoc query builder
- Introducing a first-class backend `User`/`Group` table — this issue works within the existing JWT-derived identity model unless R4's decision explicitly calls for persisting group membership
- Real-time/streaming cost updates — existing polling/on-demand refresh patterns (e.g. the "Pull Actuals" button) are preserved
- Retroactive backfill of `user_id` for invocations recorded before this feature, beyond what's needed to make the "Unattributed" bucket behave sensibly (full backfill tracked separately if warranted per R5)

Contributor guide

Open the contributing guide

Research direction

Start with backend/app/routers/costs.py and the Invocation/InvocationSession models to trace the existing agent aggregation and user_id join. Then review auth.py, AuthContext.tsx, CostDashboardPage.tsx, and the RegistryPage.tsx or CatalogPage.tsx filtering precedents. Done means per-user and per-group totals, explicit Unattributed handling, combinable filters, appropriate visibility rules, and passing backend tests plus the frontend typecheck.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
api, backend, databases, frontend, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.