Proposal: Introduce metrics reporting for lance

As a table format, Lance plays a crucial role in data storage and management for AI data. A comprehensive metrics system is essential in monitoring table formats, diagnosing health status, and performance-oriented governance. This is fully reflected in Apache Iceberg.

Apache Iceberg introduced metrics reporting1. It exposes some metadata and statistical information of core table operations (such as Scan/Commit) in the form of metrics. This ensures that these core pieces of information are not hidden beneath the iceberg.

In terms of its uses, with the external exposure of these metrics, we can do many things.

• Monitoring: Includes some indicators related to load and performance. Third parties can provide some insightful dashboards based on these indicators 24;
• Diagnosis: Based on monitoring, we can provide some incremental value. One of them is the diagnosis of some performance issues, which will assist in optimizing layout or performance;
• Optimization: Based on the provided indicators, we can optimize performance at both the read and write ends.

The issue names no files, tests, or implementation entry points. Start by reading the linked Apache Iceberg metrics-reporting documentation and then locate Lance's existing table Scan and Commit operations. Done should be defined by an agreed scope for the metrics, their exposure, and their monitoring use cases.