Adding a liveness check for consuming segments before queries are sent

Is it possible to add a check for consuming segments to only serve queries when they are close to caught up? This is similar to the consumption based status checker but in steady state for rebalances, etc: https://github.com/apache/pinot/pull/9244

Review PR #9244 and the existing consumption based status checker first. Determine how consuming-segment readiness should be checked before queries are served, including during steady-state rebalances; done means queries are withheld until segments are close to caught up.

Written by the indexing model from the issue text.