Auto-scaling and worker management

Setup auto-scaling and worker management for handling variable load.

**Requirements:**
- Configure horizontal Pod autoscaling (HPA) for Kubernetes
- Implement custom metrics for scaling decisions
- Setup worker scaling for background jobs (RabbitMQ workers)
- Configure scale-up and scale-down policies
- Add monitoring for scaling events
- Test scaling under load

**Deliverables:**
- HPA configuration for API and workers
- Custom metric definitions
- Scaling policies and thresholds
- Monitoring and alerting for scaling events
- Documentation for scaling configuration
- Load testing to verify scaling behavior

Start by locating the Kubernetes deployment manifests, API and worker entry points, RabbitMQ configuration, and existing monitoring setup; the issue names no specific files. Define the custom metrics, thresholds, scaling policies, and alerts before implementing them. Done means API and workers scale under load, scaling events are monitored, and the configuration and verification steps are documented.

Written by the indexing model from the issue text.