Data loss in offline table after tenant migration and rebalancing (no server restart)

We lose some data after migrating table to another tenant and rebalancing with downtime (no server restart). Is server restart a required step after migrating table to another tenant?

We did the following operation to move a table to another tenant
1. update tenant server in table config.
2. rebalance table with downtime.

After rebalanced, the below queries doesn't return result as expected.
`select count(0), datestr from
rta.rta.panorama_city_minute_metrics
where datestr>=‘2022-03-28’
group by 2 order by 2`

the result is as follows
`
count(*) | datestr
1761336 | 2022-03-28
1438965 | 2022-03-30
1751511 | 2022-03-31
1789216 | 2022-04-01
1796171 | 2022-04-02
1772584 | 2022-04-03
1756764 | 2022-04-04
1756555 | 2022-04-05
`

We have rerun the hive to pinot pipelines multiple times (without restarting server), the data on 03-29 is still not queryable.
Queries will return different number of records. sometime 03-30 data is also missing. In external view each servers has three servers.

i looked into the servers holding segments that match the date 2022-03-29, and it shows the following

`
qiaochu@pinotHost:~$ sudo cat /var/upinot/stream-pinot-server/dataDir/rta_panorama_city_minute_metrics_OFFLINE/rta_panorama_city_minute_metrics_1648598399_1648598399_00000/v3/metadata.properties |grep date
segment.dimension.column.names = city,completed_avg_surge,completed_avg_surge_sub_0,completed_avg_surge_sub_1,completed_trips,datestr,demand_jobs_accepted_rate_sub_0,demand_jobs_count_driver_canceled,demand_jobs_requested,driver_acceptance_rate,driver_accepts,driver_cancellation_rate,driver_offered_dispatches,eyeball_eta_sum,geodriver_accepted,geodriver_arrived,geodriver_dispatched,geodriver_driving_client,geodriver_open,gross_bookings_usd,helix_sessions,helix_sessions_surged,mean_demand_jobs_ata,mean_demand_jobs_ata_sub_0,mean_demand_jobs_ata_sub_1,mean_eta_post_request,mean_eta_post_request_sub_0,mean_eta_post_request_sub_1,mean_zero_adjusted_eyeball_eta_per_shopping_state,mean_zero_adjusted_eyeball_eta_per_shopping_state_sub_0,mean_zero_adjusted_eyeball_eta_per_shopping_state_sub_1,mean_zero_adjusted_eyeball_eta_per_shopping_state_sub_2,minute,non_requesting_non_zeroed_ss,nonzero,pct_surged_demand_jobs,requesting_sessions,shopping_sessions_v1,summary_completed_sessions,summary_driver_cancel,summary_non_stressed_nrs,summary_requesting_sessions,summary_rider_cancel,summary_stressed_nrs,summary_unfulfilled,supply_minutes,surged_demand_jobs_completed_avg_surge,surged_demand_jobs_completed_avg_surge_sub_0,surged_demand_jobs_completed_avg_surge_sub_1,surged_demand_jobs_count,surged_session_avg_surge_bb8,surged_session_avg_surge_bb8_sub_0,surged_session_avg_surge_bb8_sub_1,surged_unique_session_count_bb8_hp,time_bucket,unique_session_count_bb8_hp,zeroed_sessions,zeroed_shopping_sessions_in_sessions_stats,zeros
segment.datetime.column.names =
column.datestr.cardinality = 1
column.datestr.totalDocs = 1438965
column.datestr.dataType = STRING
column.datestr.bitsPerElement = 1
column.datestr.lengthOfEachEntry = 10
column.datestr.columnType = DIMENSION
column.datestr.isSorted = true
column.datestr.hasNullValue = false
column.datestr.hasDictionary = true
column.datestr.textIndexType = NONE
column.datestr.hasInvertedIndex = true
column.datestr.isSingleValues = true
column.datestr.maxNumberOfMultiValues = 0
column.datestr.totalNumberOfEntries = 1438965
column.datestr.isAutoGenerated = false
column.datestr.defaultNullValue = null
column.datestr.minValue = 2022-03-30
column.datestr.maxValue = 2022-03-30
`

the segment rta_panorama_city_minute_metrics_OFFLINE/rta_panorama_city_minute_metrics_1648598399_1648598399_00000 should show datestr.minValue 03-29 but it shows as 03-30.

We rerun the hive to Pinot jobs. then we restart the host, and we get the following
`
qiaochu@streampinot-prod12-dca1:~$ sudo ls -l /var/upinot/stream-pinot-server/dataDir/rta_panorama_city_minute_metrics_OFFLINE/rta_panorama_city_minute_metrics_1648598399_1648598399_00000/v3/
total 143184
-rwx------ 1 upinot upinot 146537680 Apr 4 20:34 columns.psf
-rwx------ 1 upinot upinot 16 Apr 4 20:34 creation.meta
-rwx------ 1 upinot upinot 13571 Apr 4 20:34 index_map
-rwx------ 1 upinot upinot 60446 Apr 4 20:34 metadata.properties
qiaochu@streampinot-prod12-dca1:~$ sudo cat /var/upinot/stream-pinot-server/dataDir/rta_panorama_city_minute_metrics_OFFLINE/rta_panorama_city_minute_metrics_1648598399_1648598399_00000/v3/metadata.properties |grep date
segment.dimension.column.names = city,completed_avg_surge,completed_avg_surge_sub_0,completed_avg_surge_sub_1,completed_trips,datestr,demand_jobs_accepted_rate_sub_0,demand_jobs_count_driver_canceled,demand_jobs_requested,driver_acceptance_rate,driver_accepts,driver_cancellation_rate,driver_offered_dispatches,eyeball_eta_sum,geodriver_accepted,geodriver_arrived,geodriver_dispatched,geodriver_driving_client,geodriver_open,gross_bookings_usd,helix_sessions,helix_sessions_surged,mean_demand_jobs_ata,mean_demand_jobs_ata_sub_0,mean_demand_jobs_ata_sub_1,mean_eta_post_request,mean_eta_post_request_sub_0,mean_eta_post_request_sub_1,mean_zero_adjusted_eyeball_eta_per_shopping_state,mean_zero_adjusted_eyeball_eta_per_shopping_state_sub_0,mean_zero_adjusted_eyeball_eta_per_shopping_state_sub_1,mean_zero_adjusted_eyeball_eta_per_shopping_state_sub_2,minute,non_requesting_non_zeroed_ss,nonzero,pct_surged_demand_jobs,requesting_sessions,shopping_sessions_v1,summary_completed_sessions,summary_driver_cancel,summary_non_stressed_nrs,summary_requesting_sessions,summary_rider_cancel,summary_stressed_nrs,summary_unfulfilled,supply_minutes,surged_demand_jobs_completed_avg_surge,surged_demand_jobs_completed_avg_surge_sub_0,surged_demand_jobs_completed_avg_surge_sub_1,surged_demand_jobs_count,surged_session_avg_surge_bb8,surged_session_avg_surge_bb8_sub_0,surged_session_avg_surge_bb8_sub_1,surged_unique_session_count_bb8_hp,time_bucket,unique_session_count_bb8_hp,zeroed_sessions,zeroed_shopping_sessions_in_sessions_stats,zeros
segment.datetime.column.names =
column.datestr.cardinality = 1
column.datestr.totalDocs = 1744810
column.datestr.dataType = STRING
column.datestr.bitsPerElement = 1
column.datestr.lengthOfEachEntry = 10
column.datestr.columnType = DIMENSION
column.datestr.isSorted = true
column.datestr.hasNullValue = false
column.datestr.hasDictionary = true
column.datestr.textIndexType = NONE
column.datestr.hasInvertedIndex = true
column.datestr.isSingleValues = true
column.datestr.maxNumberOfMultiValues = 0
column.datestr.totalNumberOfEntries = 1744810
column.datestr.isAutoGenerated = false
column.datestr.defaultNullValue = null
column.datestr.minValue = 2022-03-29
column.datestr.maxValue = 2022-03-29
`
the query result correct result after restarting the host.

We have two questions
1. what's the root cause that 03-29 data are missing in the table.
2. Why segment with the 1648598399_1648598399 timestamp in the name, holding the 03-30 day's data before we restart server?

Start by reviewing the tenant migration and downtime rebalancing steps, then compare the segment's metadata.properties before and after the host restart. Reproduce the missing 2022-03-29 query results and inspect the named segment path and Hive-to-Pinot pipeline behavior. Done means identifying the cause of the stale date metadata and explaining the timestamp discrepancy.

Written by the indexing model from the issue text.