ASOF join support / Specialize Range Joins
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In many timeseries workloads, the need to join one record to another based on recency or ordering is quite prevalent. This can be with two sensors that maintain different sampling rates, in handling market data (such as situations where you want to find the most recent quote for a given trade), or any other scenario where an ordering of events can be applied. In these situations, it is quite frequent that one set of data is at a *much* larger scale than the other, typically starting at an order of magnitude. It is also quite common that you would want to actually store your data *in* the appropriate order so as to minimize the effort when performing this kind of join operation (though that is not strictly the problem itself).
**Describe the solution you'd like**
Essentially the solution implemented by [Clickhouse ](https://clickhouse.tech/docs/en/sql-reference/statements/select/join/#asof-join-usage) and [KDB](https://code.kx.com/q/ref/aj/) (please note that the Clickhouse solution allows for the full breadth of ordering conditions for closest match).
**Describe alternatives you've considered**
There is a general [subquery solution](https://dba.stackexchange.com/questions/116424/left-join-on-closest-date) that can be used to achieve the desired outcome, but it is typically not performant and can be fairly awkward to express. Provided implementation of #141 and sufficient query optimization of the subquery solution into sort-merge join, it may not be necessary as direct syntax.
**Additional context**
- This likely depends on #141
- Related to https://github.com/apache/datafusion/issues/8393
Contributor guide
Research direction
Start by reviewing the dependency on #141 and the related issue #8393, then compare the requested behavior with the linked ClickHouse and KDB references. Done means DataFusion can express ASOF or specialized range joins for ordered, unevenly sampled data, including the intended ordering conditions and acceptable alternatives such as an optimized subquery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100