[FEA]: Prometheus Performance Metrics Exposure for NV-Ingest Pipeline

Is this a new feature, an improvement, or a change to existing functionality?

Improvement

How would you describe the priority of this feature request

Significant improvement

Please provide a clear description of problem this feature solves

This feature is essential for effective monitoring, troubleshooting, and autoscaling in production environments. Currently, NV-Ingest and its NIM services do not expose detailed performance metrics in a way that can be easily consumed by monitoring tools like Prometheus. This lack of visibility makes it difficult to:

• Monitor system health and performance in real time
• Identify bottlenecks within the ingestion pipeline
• Configure Kubernetes Horizontal Pod Autoscaler (HPA) rules based on actual workload and resource usage

Describe the feature, and optionally a solution or implementation and any alternatives

Expose key operational metrics (such as latency, throughput, queue size, error rates, and resource utilization) for the entire NV-Ingest pipeline and each NIM service via Prometheus-compatible endpoints.

Implementation Suggestion:

• Integrate Prometheus client libraries into NV-Ingest and all NIM services
• Expose some sort of central /metrics endpoints for the entire pipeline or each service
• Ensure metrics are labeled for per-service and per-stage granularity
• Leverage a service that aggregates metrics across each service and can be scraped by Prometheus service monitors

Alternatives Considered:

• Relying on application logs (less efficient, harder to automate scaling)

Additional context

• This feature aligns with best practices for cloud-native observability and autoscaling.
• It enables proactive scaling and alerting, improving reliability and operational efficiency.
• Granular metrics are especially important in complex, multi-service pipelines where load may not be evenly distributed.

The issue names the NV-Ingest pipeline and its NIM services but no files or tests. Start by mapping those services and checking whether any Prometheus-compatible endpoints or monitoring components already exist. Done would require an agreed metric set, labeling granularity, and scrapeable endpoints suitable for pipeline and service monitoring.