[FEATURE] Introduce Optimizer Service for table maintenance workflows

Describe the feature

Introduce a long-running Optimizer Service for Apache Gravitino Table Maintenance Service workflows.

The service should provide REST APIs for optimizer operations that are currently executed through local CLI commands, including:

• update-statistics
• append-metrics
• submit-strategy-jobs
• monitor-metrics
• list-table-metrics
• list-job-metrics

The service should support asynchronous task execution, request IDs, task status query, cancellation, and structured results.

Motivation

The current optimizer workflow is mainly local CLI driven. This makes it difficult to support centralized task tracking, retries, concurrency control, audit logging, service-level metrics, and structured automation APIs.

A long-running Optimizer Service would let users submit table maintenance work, disconnect from the CLI process, and later query task status and results through stable REST APIs.

Describe the solution

Add an independent Optimizer Service in the optimizer distribution.

High-level scope:

• Add OptimizerServiceServer as a standalone optimizer service entry point.
• Add a shared task runtime for async execution and task state management.
• Add REST APIs for updater, recommender, monitor, and metrics-query operations.
• Keep existing CLI commands and add service mode so the CLI can call the service when configured.
• Preserve local CLI execution as the default for backward compatibility.
• Keep submit-update-stats-job integrated with the existing Gravitino job framework.

A detailed design document is proposed in design-docs/optimizer-service-design.md.

Additional context

This is intended as an incremental evolution of the alpha Table Maintenance Service (Optimizer). The initial implementation can use an in-memory task store, with persistent task storage and stronger multi-node support added later.

Start with design-docs/optimizer-service-design.md and inspect the existing optimizer distribution and local CLI commands for update-statistics, append-metrics, submit-strategy-jobs, monitor-metrics, list-table-metrics, and list-job-metrics. Done means a standalone OptimizerServiceServer exposes these operations through REST with asynchronous task tracking, request IDs, status queries, cancellation, and structured results while preserving local CLI behavior.