Arrow Flight Endpoint for Pinot

In addition to being used as the in-memory and wire columnar format in few compute engines, Arrow is also commonly used for data sharing between JVM and non JVM systems without SerDe overhead. So python users working with Pandas and other analytical libraries can consume arrow in-memory format generated by JVM based engine.

See this example on how PySpark uses Arrow - https://kontext.tech/column/spark/370/improve-pyspark-performance-using-pandas-udf-with-apache-arrow

Arrow flight is the optimized wire protocol for network transfer of columnar record batches (think of as alternative to JDBC and ODBC protocol). The wire format is same as in-memory format. So when both endpoints are using Arrow, Flight protocol can be used to efficiently send result data from Pinot as Arrow record batches to say a Python client which can continue to do additional processing on it.

No files, tests, or entry points are named. Start by locating Pinot's existing query endpoints and the project’s Arrow-related integration points, then determine the protocol and API boundaries needed for an Arrow Flight endpoint. Done means Pinot can expose query results as Arrow record batches consumable by a Python client.

Written by the indexing model from the issue text.