Support Adaptive Encoding
- Dominant language
- Java
- Stars
- 203
- Forks
- 104
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 33
Description
Currently, TsFile uses a fixed encoding algorithm for each data type. For example, floating-point data may use Gorilla encoding.
However, data characteristics vary across datasets, and a fixed encoding algorithm may not always perform well. In some cases, it may even increase the encoded data size. We propose introducing an adaptive encoding mechanism that dynamically selects a more suitable encoding algorithm based on the actual data characteristics or encoding effectiveness.
Falling back to PLAIN when the current encoding is unsuitable can be one adaptive option, but it should not be the only one. The selection strategy, evaluation criteria, and switching granularity can be discussed separately.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by locating TsFile’s fixed encoding path for each data type and reviewing the existing encoding implementations; the work is not ready to start until the selection strategy, evaluation criteria, switching granularity, and completion criteria are agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100