✨ add X-Ray tracing integration

Summary

Add AWS X-Ray distributed tracing support for enhanced observability, enabling end-to-end request visibility across the zae-limiter stack.

Motivation

X-Ray tracing provides:

• Root cause analysis - Trace slow requests through Lambda, DynamoDB, and client code
• Performance optimization - Identify bottlenecks in acquire/release operations
• Cross-service correlation - Connect rate limiter traces to upstream application traces
• Production debugging - Debug issues without adding ad-hoc logging

Phased Implementation

Phase 1: Lambda Active Tracing (No blockers)

Enable X-Ray at the infrastructure level without SDK dependencies:

• Add TracingConfig: Mode: Active to CloudFormation Lambda function
• Update IAM role with xray:PutTraceSegments and xray:PutTelemetryRecords permissions
• Add enable_tracing option to StackOptions
• CLI: --enable-tracing / --no-tracing flags for deploy command
• AWS E2E tests (locally run with --run-aws) verifying:
  • Lambda tracing configuration
  • IAM permissions
  • Actual X-Ray traces are submitted when Lambda executes

This phase can be implemented immediately - it only requires CloudFormation changes.

Phase 2: SDK Instrumentation (Blocked by #154)

Add client-side and Lambda instrumentation with aws-xray-sdk:

• Add aws-xray-sdk as optional dependency (pip install zae-limiter[xray])
• Patch boto3 clients in Repository class when X-Ray SDK is available
• Instrument DynamoDB calls: GetItem, Query, TransactWriteItems, etc.
• Add custom subsegments for acquire(), release(), adjust() operations
• Graceful degradation when SDK not installed

Blocked by #154 - Lambda packaging must support dependencies beyond boto3 before we can include the X-Ray SDK in the aggregator Lambda.

Phase 3: CI Test Coverage (Blocked by #189)

Add automated testing for X-Ray trace verification in CI:

• E2E tests verifying traces appear in X-Ray console (run in GitHub Actions)
• Integration with AWS E2E workflow

Blocked by #189 - X-Ray is not supported in LocalStack Community Edition (requires Enterprise). We need real AWS tests in CI before we can properly test X-Ray trace output.

Implementation Details

CloudFormation Changes (Phase 1)

# cfn_template.yaml additions
Parameters:
  EnableTracing:
    Type: String
    Default: 'false'
    AllowedValues: ['true', 'false']
    Description: Enable X-Ray tracing for the aggregator Lambda

Conditions:
  TracingEnabled: !Equals [!Ref EnableTracing, 'true']

Resources:
  AggregatorFunction:
    Properties:
      TracingConfig:
        Mode: !If [TracingEnabled, Active, PassThrough]

  AggregatorRole:
    Properties:
      Policies:
        - PolicyName: XRayAccess
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  - xray:PutTraceSegments
                  - xray:PutTelemetryRecords
                Resource: '*'

StackOptions Addition (Phase 1)

@dataclass(frozen=True)
class StackOptions:
    # ... existing fields ...
    enable_tracing: bool = False  # Enable X-Ray tracing

Optional SDK Integration (Phase 2)

# In repository.py or a new tracing.py module
try:
    from aws_xray_sdk.core import xray_recorder, patch
    XRAY_AVAILABLE = True
except ImportError:
    XRAY_AVAILABLE = False

def instrument_clients():
    """Patch boto3 clients for X-Ray tracing if SDK is available."""
    if XRAY_AVAILABLE:
        patch(['boto3'])

API/CLI Parity

Feature	API	CLI
Enable tracing	StackOptions(enable_tracing=True)	--enable-tracing
Disable tracing	StackOptions(enable_tracing=False)	--no-tracing (default)

Dependencies

Blocked By

• #154 - Investigate Lambda packaging alternatives to boto3-only restriction (required for Phase 2)
• #189 - Add AWS E2E tests with GitHub OIDC authentication (required for Phase 3)

Related

• #132 - CloudFormation creates IAM roles (should include X-Ray permissions in App/Admin roles)
• #152 - IAM policy command (should include X-Ray permissions for deploy role)
• #187 - Analytics dashboard (X-Ray can complement CloudWatch-based analytics)
• #38 - Monitoring guide (documentation reference, closed)

Documentation Updates

• docs/monitoring.md: Replace "Future Enhancement" placeholder with full X-Ray setup guide
• docs/infra/deployment.md: Add --enable-tracing flag documentation
• CLAUDE.md: Add X-Ray to StackOptions documentation

Testing Strategy

Phase 1 Testing

Phase 1 can be fully tested with a combination of mocked and real AWS tests:

Test	Backend	What It Verifies
Unit	moto	StackOptions(enable_tracing=True) validation
Unit	moto	CLI --enable-tracing flag parsing
Integration	LocalStack	CloudFormation deploys with EnableTracing parameter
Integration	LocalStack	IAM role includes X-Ray permissions when enabled
E2E	Real AWS	Lambda has TracingConfig.Mode: Active when enabled
E2E	Real AWS	IAM role has X-Ray permissions
E2E	Real AWS	X-Ray traces are actually submitted after Lambda invocation

AWS E2E tests are run locally with pytest --run-aws and verify the actual AWS resources are configured correctly and traces are generated.

Phase 2 Testing

Test	Backend	What It Verifies
Unit	moto	SDK instrumentation logic
Unit	moto	Graceful degradation when SDK not installed

Phase 3 Testing (Requires #189)

Test	Backend	What It Verifies
E2E	Real AWS (CI)	Traces appear in X-Ray console
E2E	Real AWS (CI)	End-to-end trace correlation

Note: X-Ray is not available in LocalStack Community Edition. Phase 3 testing requires the AWS E2E infrastructure from #189.

Cost Considerations

X-Ray pricing (as of 2024):

• Free tier: 100,000 traces recorded/month, 1M traces scanned/month
• Paid: $5.00 per million traces recorded, $0.50 per million traces scanned

For most rate limiting deployments, X-Ray costs should be minimal.

Acceptance Criteria

Phase 1

• CloudFormation template supports EnableTracing parameter
• IAM role includes X-Ray permissions when tracing enabled
• StackOptions includes enable_tracing option
• CLI deploy supports --enable-tracing / --no-tracing
• Unit tests for StackOptions validation and CLI flags
• Integration tests verify CloudFormation deploys with tracing config
• AWS E2E tests verify Lambda tracing config and IAM permissions (run locally with --run-aws)
• AWS E2E tests verify X-Ray traces are actually submitted when Lambda executes

Phase 2

• aws-xray-sdk is an optional dependency (zae-limiter[xray])
• DynamoDB calls are instrumented when SDK is installed
• Custom subsegments for acquire/release operations
• Graceful degradation when SDK not installed

Phase 3

• E2E tests verify traces in X-Ray console (run in CI)
• Documentation updated with X-Ray setup guide
• Monitoring guide placeholder replaced with full documentation

References

• AWS X-Ray SDK for Python
• Instrumenting Python with X-Ray
• Lambda X-Ray Tracing
• DynamoDB X-Ray Integration

Start with cfn_template.yaml, StackOptions, and the deploy CLI entry point to scope Phase 1. Review the existing unit and integration tests, then run the AWS checks with pytest --run-aws. Done means the tracing option and flags, CloudFormation configuration, IAM permissions, and documented monitoring setup are covered by the stated tests.