apache / apache/pinot

[Tracker] Time Series Engine GA

Closed
#15,974 4 comments 0 reactions 1 assignee Claimed by @shauryachats View on GitHub
stale timeseries-engine timeseries-general-availability
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Tracker task for pending items for the Time Series Engine.

- [x] Add integration tests for Time Series Engine. #15960
- [x] Remove reliance on table-type suffix. Presently users need to specify table name with type. This is because of the logic in `TableScanVisitor` which calls routing manager directly based on the user provided table name.
- [x] Re-think HTTP API. We had gone with a Prometheus compatible API, but I don't see any value in it. Moreover, there are some Pinot query related stats that we would like to return in the response. Maybe we should consider returning in the Broker response format itself?
- [x] Enable standard auth integrations
- [x] Integrate with query event listeners
- [x] Add support for query options
- [x] Support Explain Plans (this is dependent on HTTP API)
- [x] Add docs in the Apache Pinot wiki
- [ ] Support for Hybrid Tables
- [x] Evaluate merge with Multistage Engine
- [ ] Fix the plan serialization scheme. Currently it relies on a hacky Jackson based approach.
- [x] Think about Exemplar support.

... more to be added soon

# Footnotes

## Merging with MSE

Right now we have very little custom code for the Time Series Engine left. But still, we should see if we can integrate with the MSE Runtime to take advantage of stats propagation, plan serde, etc. We recently also contributed MSE Lite Mode with runInBroker query option, which is similar to the execution model we are using in the Time Series Engine, which would facilitate the integration.

At a high-level, I think these are the main things involved in merging TSE with MSE:

1. Adding new `PlanNode` types for running chains of `BaseTimeSeriesPlanNode`. Then in the time series planner could simply output a tree of `PlanNode` that could be handled in theory the same way by the MSE.
2. Adding new MSE Operator corresponding to the PlanNode mentioned above.
3. Handling compilation of the new PlanNode and the leaf stage in the server.
4. Switching to MseBlock for data transfers. We could still use the TimeSeriesBlock for in-memory computations and allowing users to implement custom operators.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.