Controller should reject out-of-retention segments

Controller currently has no guardrail over ingestion of out-of-retention segments. We've encountered this issue in production due to incorrectly configured ingestion jobs.
Currently only Retention Manager handles their clean up. Adding this validation to Controller segment upload APIs can solve following issues.
1. Reduces load of invalid segments from servers
2. Retention Manager and servers can have a race condition leaving out-of-retention segments temporarily in ERROR state (depicted in diagram)

Start by tracing the Controller segment upload APIs and the Retention Manager cleanup path described in the issue. Determine how out-of-retention segments are identified and where upload validation belongs; done means invalid segments are rejected before ingestion without leaving the reported ERROR-state race.

Written by the indexing model from the issue text.