GetRecord.IteratorAgeInMilliseconds metrics not being reflected and data not being queried in Table

Table Config
```{
"tableName": "workflowEvents",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "eventTimestamp",
"timeType": "MILLISECONDS",
"schemaName": "workflowEvents",
"replicasPerPartition": "1",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "365",
"segmentPushType": "APPEND"
},
"tenants": {
"broker":"DefaultTenant",
"server":"DefaultTenant"
},
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kinesis",
"stream.kinesis.topic.name": "prod-rel-cdp-dl-workflow-metrics-stream",
"region": "us-east-1",
"shardIteratorType": "LATEST",
"stream.kinesis.consumer.type": "lowlevel",
"stream.kinesis.fetch.timeout.millis": "30000",
"stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory",
"realtime.segment.flush.threshold.size": "5000000",
"realtime.segment.flush.threshold.time": "1d"
}
},
"upsertConfig": {
"mode": "FULL"
},
"routing": {
"instanceSelectorType": "strictReplicaGroup"
},
"metadata": {
"customConfigs": {}
}
}
```

- I upgraded pinot to 0.9.1 version from 0.8.0 last month and after that I started seeing GetRecord.IteratorAgeInMilliseconds metrics displaying in correct values
- Yesterday I observed the data stopped being displayed in the Table from the Query Console even though the GetRecords metrics showed that records were read but where not read from the Table.There were some new data fields that were added to the record but the schema was not updated
- I decided to change Table config to AT_SEQUENCE_NUMBER and I did not see the changes immediately but this morning I was able to see those records but post that when records were added this morning they were not being reflected in the table
- I decided to switch the Table config back to LATEST.I also saw that the old segments got terminated and new segments were created but the records were still not reflected in the Table
So both the data issue and GetRecord.IteratorAgeInMilliseconds still persist

Start by reproducing the reported behavior with the supplied REALTIME table configuration, comparing GetRecords and GetRecord.IteratorAgeInMilliseconds with Query Console results. Check the effect of the missing schema fields and switching shardIteratorType between LATEST and AT_SEQUENCE_NUMBER; done means newly ingested records are queryable and the iterator-age metric reflects ingestion lag.

Written by the indexing model from the issue text.