Add metrics and logging for key route affinity decisions

Summary

Add optional metrics and debug logging to help operators understand and tune key route affinity behavior in production environments.

Background

Key route affinity silently makes routing decisions based on partition key hashing. Without visibility into these decisions, operators cannot:

• Verify that affinity is working as expected
• Identify tables with missing PK configuration (falling back to round-robin)
• Monitor the distribution of requests across nodes
• Debug unexpected routing behavior

Proposed Metrics

Counters

Metric	Description
alternator.affinity.requests.total	Total requests processed by interceptor
alternator.affinity.requests.affinity	Requests routed using key affinity
alternator.affinity.requests.roundrobin	Requests falling back to round-robin
alternator.affinity.pk.cache.hits	Partition key name cache hits
alternator.affinity.pk.cache.misses	Partition key name cache misses
alternator.affinity.pk.discovery.triggered	PK discovery attempts triggered
alternator.affinity.pk.discovery.success	Successful PK discoveries
alternator.affinity.pk.discovery.failed	Failed PK discoveries

Gauges

Metric	Description
alternator.affinity.pk.cache.size	Number of cached table PK mappings
alternator.affinity.pk.failed.tables	Tables in failure cooldown

Labels/Tags

• table - Table name (for per-table breakdown)
• mode - Affinity mode (RMW, ANY_WRITE)
• reason - Fallback reason (no_pk_info, not_qualifying_request, etc.)

Proposed Logging

DEBUG Level

Key affinity applied: table=users, pk=user_123, target=192.168.1.5:8000
Key affinity skipped: table=users, reason=request_not_qualifying (simple PutItem in RMW mode)
Key affinity skipped: table=orders, reason=pk_not_cached (discovery triggered)

INFO Level (on significant events)

PK discovery completed: table=orders, pk_attribute=order_id
PK discovery failed: table=unknown_table, reason=ResourceNotFoundException

Implementation Options

Option 1: Micrometer Integration

Use Micrometer for metrics, allowing integration with Prometheus, CloudWatch, etc.

public class KeyRouteAffinityInterceptor implements ExecutionInterceptor {
    private final MeterRegistry meterRegistry;
    private final Counter affinityCounter;
    private final Counter roundRobinCounter;
    // ...
}

Pros: Industry standard, wide ecosystem support
Cons: Adds dependency, may be overkill for some users

Option 2: Callback Interface

Provide a callback interface for users to implement their own metrics collection.

public interface AffinityMetricsCallback {
    void onAffinityApplied(String table, String pkValue, URI targetNode);
    void onAffinitySkipped(String table, String reason);
    void onPkDiscovery(String table, boolean success);
}

Pros: No dependencies, flexible
Cons: More work for users

Option 3: JUL Logging Only

Enhance existing java.util.logging with structured debug output.

Pros: No new dependencies
Cons: Limited metrics capabilities

Acceptance Criteria

• Metrics can be enabled/disabled via configuration
• Logging at DEBUG level provides request-level visibility
• Metrics have minimal performance impact when disabled
• Documentation explains available metrics and how to use them
• Example integration with at least one metrics backend (e.g., Prometheus)

Files to Modify

• src/main/java/com/scylladb/alternator/keyrouting/KeyRouteAffinityInterceptor.java
• src/main/java/com/scylladb/alternator/keyrouting/KeyRouteAffinityConfig.java
• src/main/java/com/scylladb/alternator/keyrouting/PartitionKeyResolver.java

Open Questions

1. Should metrics be opt-in (disabled by default) or opt-out?
2. Which metrics library (if any) should be supported?
3. Should we support multiple metrics backends or just one?

Start by reading KeyRouteAffinityInterceptor.java, KeyRouteAffinityConfig.java, and PartitionKeyResolver.java to understand the existing affinity and PK discovery flow. Resolve the open questions about configuration, metrics integration, and logging before implementation. Done means configurable metrics and debug logging with low disabled overhead, documentation, and an example backend integration.