[TASK] Implement Retry Analytics with Decline Normalization
- Dominant language
- Rust
- Stars
- 43.7k
- Forks
- 5.1k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 205
Description
## Summary
Build retry analytics APIs that normalize decline reasons across Payment Service Providers (PSPs) using standardized error codes. This enables merchants to understand true decline distributions and optimize retry strategies.
## Goal
- Add `standardised_code` and `error_category` fields to Kafka events and ClickHouse schema
- Build GSM cache module for historical data fallback
- Create 7 new analytics API endpoints under `/analytics/v1/metrics/retry/`
- Enable error-level retry insights (success rates, optimal timing, cross-PSP comparison)
## Technical Approach
### 1. Kafka Events Enhancement
**Files:**
- `crates/router/src/services/kafka/payment_attempt_event.rs`
- `crates/router/src/services/kafka/payment_attempt.rs`
Add `standardised_code: Option` and `error_category: Option` fields to both v1 and v2 KafkaPaymentAttemptEvent structs.
Update `from_storage()` to map from nested error_details.unified_details structure.
### 2. ClickHouse Schema Update
**File:** `crates/analytics/docs/clickhouse/scripts/payment_attempts.sql`
Add columns to payment_attempts table:
```sql
standardised_code LowCardinality(Nullable(String)),
error_category LowCardinality(Nullable(String)),
INDEX idx_standardised_code_bloom standardised_code TYPE bloom_filter GRANULARITY 3,
INDEX idx_error_category_bloom error_category TYPE bloom_filter GRANULARITY 3,
```
**Note:** Materialized Views must be dropped and recreated to include new columns.
### 3. GSM Cache Module
**New file:** `crates/analytics/src/gsm_cache.rs`
- In-memory cache with HashMap
- Load from PostgreSQL on startup
- 10-minute periodic refresh
- Fallback lookup when ClickHouse columns are NULL
### 4. Analytics Crate Enhancement
**New directory:** `crates/analytics/src/retry_analytics/`
Files to create:
- `mod.rs` - Module exports
- `types.rs` - Request/response types, RetryDimension, RetryMetrics enums
- `filters.rs` - Filter query implementations
- `metrics.rs` - Metric implementations using QueryBuilder
- `core.rs` - Main query orchestration
**QueryBuilder Pattern:**
```rust
let mut query_builder: QueryBuilder = QueryBuilder::new(AnalyticsCollection::Payment);
query_builder.add_select_column(dim).switch()?;
query_builder.add_select_column(Aggregate::Count { ... }).switch()?;
query_builder.add_filter_clause("status", AttemptStatus::Failure).switch()?;
query_builder.add_group_by_clause(dim).switch()?;
```
**Note:** Retry correlation queries requiring argMin/argMax should use raw SQL execution through ClickHouse client (known QueryBuilder limitation).
### 5. API Endpoints
**File:** `crates/router/src/analytics.rs`
Add 7 new endpoints:
- `POST /analytics/v1/metrics/retry/normalized-declines`
- `POST /analytics/v1/metrics/retry/connector-decline-matrix`
- `POST /analytics/v1/metrics/retry/retry-effectiveness`
- `POST /analytics/v1/metrics/retry/retry-by-connector`
- `POST /analytics/v1/metrics/retry/retry-delay-analysis`
- `POST /analytics/v1/metrics/retry/decline-classification`
- `POST /analytics/v1/metrics/retry/retry-recommendations`
### 6. GSM Cache Integration
**File:** `crates/router/src/core/gsm.rs`
Integrate GSM cache for fallback lookups when ClickHouse columns are NULL.
## Subtasks
- [ ] Add standardised_code and error_category to Kafka events
- [ ] Update ClickHouse schema with new columns and bloom filter indexes
- [ ] Create GSM cache module with periodic refresh
- [ ] Implement retry_analytics module with types, filters, metrics
- [ ] Create 7 API endpoints with proper routing
- [ ] Implement retry correlation using raw SQL with argMin/argMax
- [ ] Add GSM cache integration for NULL value fallback
- [ ] Write unit tests for GSM cache and metrics
- [ ] Write integration tests for API endpoints
## Acceptance Criteria
- [ ] All 7 API endpoints return data with p95 latency < 2 seconds
- [ ] Kafka events include standardised_code and error_category
- [ ] ClickHouse queries use bloom filter indexes efficiently
- [ ] GSM cache refreshes every 10 minutes
- [ ] NULL standardised_code values fallback to GSM cache lookup
## Edge Cases
- Missing GSM mappings → fallback to raw error values
- Cache staleness → document 10-minute refresh window
- Historical data → NULL columns trigger GSM lookup
- Query performance → enforce time-range constraints
Contributor guide
Research direction
Start by reading the listed Kafka event files, the ClickHouse schema, and crates/router/src/analytics.rs to map the existing analytics flow. Break the work into the Kafka, schema, GSM cache, retry_analytics, routing, and test subtasks. Done means all seven endpoints and listed acceptance criteria are implemented, including cache fallback, refresh behavior, performance constraints, and tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, kafka, postgresql, rust
- Domain
- api, backend, data-engineering, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100