Enhance Data Ingestion Engine

Motivation:
- Offline segment generation and real-time in-memory consuming to on-disk committed segment conversion are very similar process, and they are both achieved by the same segment creation engine. We should try to unify the pipeline so that most of the features can be shared for both offline and real-time tables, including sorting data on a given column.
- Currently offline segment generation scans the records twice (record transformations also happen twice), which has low performance and is not friendly to streaming input.
- Currently real-time segment conversion reads the raw records from the segment, which can be significantly optimized to save both CPU and memory cost because all the useful information is already stored in the forward index.

End goal:
- Segment generation should purely based on the table config and schema, no extra parameters should be accepted (there is no way to pass extra parameters for real-time segment conversion).
- Offline data ingestion should only scans the records once and store all the information in memory (similar to real-time consuming segment).
- Offline data ingestion should support sorting segment on a given column (same as real-time segment conversion).
- Real-time segment conversion should use much less system resources and have less impact to queries.
- Design an interface (close to the idea of the stats collector) to store all the column data. Offline data ingestion pipeline creates them by scanning the data; real-time data ingestion pipeline creates them by wrapping around the in-memory forward-index, inverted-index, dictionary etc. The segment creation engine directly works on the new interface to create the segment so that offline and real-time pipeline can share the features of the segment creation engine.
- Data ingestion engine should support the following types of indexes:
- Dictionary
- Forward Index (both dictionary encoded & raw)
- Inverted Index
- Null Value Vector
- Text Index
- Star-Tree Index (v2)
- (Bloom Filter is in memory only right now)

Start by mapping the existing offline segment-generation and real-time segment-conversion pipelines, then compare how they build dictionaries, forward and inverted indexes, null-value vectors, text indexes, and Star-Tree indexes. Define the shared column-data interface and verify that offline ingestion scans records once, sorting is supported, and real-time conversion avoids rereading raw records. The issue names no files or tests, so repository exploration is required.

Written by the indexing model from the issue text.