apache / apache/pinot

Make Timestamp index an actual index

Open
#10,633 1 comment 3 reactions 1 assignee Claimed by @gortiz View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Timestamp indexes are explained as indexes to users (see the [documentation](https://docs.pinot.apache.org/basics/indexing/timestamp-index)) but they are not implemented as actual indexes. Instead, they are a syntactic sugar that:
- Modifies (see [TimestampIndexUtils](https://github.com/apache/pinot/blob/44fe6940e19cbf29d2f5f606124c2f6b77e85360/pinot-spi/src/main/java/org/apache/pinot/spi/utils/TimestampIndexUtils.java#L128))the TableConfig and Schema in order to:
- Add one new column per granularity.
- Add a range index on each column.
- Applies some query override in order to use the new columns when available. I didn't find the code where this is applied, but given that `FieldConfig.IndexType.TIMESTAMP` is not read, I guess that the query engine looks for columns with specific pattern names as a general optimization.

This is not ideal and we should correctly define Timestamp _indexes_. Options are:
1. We can either accept that they are not actual indexes and pull them somewhere else. They would still be configured in `tableConfig.fieldConfigList.timestampConfig` but we will not call them _indexes_. We would need to classify them as a new concept.
2. Or we make them actual indexes.

The first approach mainly implies to change documentation and keep the current implementation.
This will make things less confusing, but the current implementation is still difficult to maintain and/or not very efficient:
- Usually implementations do not actually modify the `TableConfig` and `Schema`.
- Adding more columns may be problematic in terms of storage. If the table is stored by the timestamp, new columns forward indexes should be small in size, but if it is not, the these new columns can be quite large in bytes.

Therefore we propose to make them actual indexes.
This implies to create their own IndexType, IndexCreator, IndexReader, IndexHandler... but also configure them in the `tableConfig.fieldConfigList.indexes` section and store them in a single buffer instead of being an alias on top of artificial columns.

## Configuration

Like other indexes, we will add a new way to configure Timestamp indexes.
To keep backward compatibility, timestamp indexes defined in `tableConfig.fieldConfigList.timestampConfig` will continue to be supported.
But the new and recommended way to define them would be something like:

```js
//tableconfig
{
fieldConfigList: [{
name: "columnName"
indexes: {
"timestamp": {
...timestampConfig // same object we had in timestampConfig but adding the standard "disabled" property
}
}
}]
}
```

Since `index-spi` was merged, timestamp _indexes_ are the only one that cannot be defined in the `indexes` section.
To let them be configured in the `indexes` section seems to be something easy to implement.

## Index implementation
The idea here would be to implement this kind of index as a normal index, which means to store all information in a single buffer and do not create new columns.

At the end of the day a Timestamp index contains N different range indexes. Right now they are implemented as N different columns with 1 range index each. The idea here would be to store the N range indexes in 1 single buffer.
Each range index should have the same size of the one used in the current solution, but we won't need to add one forward index per granularity.

At query time, we need to add rules like we have in other indexes and apply it if the operation is compatible. The current optimizations applied for timestamp indexes are:
1. GROUP BY: functions like dateTrunc('DAY', ts) will be translated to use the underly column $ts$DAY to fetch data.
2. PREDICATE: range index is auto-built for all granularity columns.

The predicate optimization can be applied in the same way with the proposed actual index type.
The group by may be tricky and we may need to think more about them, but at worst, users can decide to create their own virtual columns like they do to optimize similar patterns with other types.

This change may be more difficult to implement and require to invest more time than the change in the configuration. If we decide that this is the solution we would like to have at long term, we can start implementing the changes in the configuration in order to provide an uniform way to configure all indexes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.