SegmentGeneratorTaskRunner doesn't take IngestionConfig into consideration

See: https://github.com/apache/pinot/blob/611f3b11b1336bc9f426ead5ba908a9632a50fd9/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common/src/main/java/org/apache/pinot/plugin/ingestion/batch/common/SegmentGenerationTaskRunner.java#L105.

By the time SegmentGeneratorConfig is created.
1. it is not using `IngestionUtils.generateSegmentGeneratorConfig` to generate segmentGeneratorConfig.
2. even if so. the SegmentNameGenerator is created without the knowledge of the tableConfig.
Thus if we set the segmentNameGenerator config in tableconfig.ingestionConfig.BatchIngestconfig.BatchConfigMap. it is not honored.

I was wondering if there's a general rule of thumb on what to expect when users are setting the configuration.

For example in this case when using the ingestion plugin. setting in tableConfig directly is useless and is not doing what the user wanted. which makes it hard to debug/trace because nothing fails.

Proposal
===

From my point of view we should have a rule of thumb.

If one setting/config-key is provided in multiple places, then:
* plugin configuration > table config > pinot default.

Start at SegmentGenerationTaskRunner.java around line 105 and trace how SegmentGeneratorConfig and SegmentNameGenerator are created, then inspect IngestionUtils.generateSegmentGeneratorConfig. Confirm the intended precedence between plugin, table, and default settings, and consider the work complete when the tableConfig ingestion settings are honored and the behavior is covered by relevant tests.

Written by the indexing model from the issue text.