apache / apache/datafusion

ASOF join support / Specialize Range Joins

Closed
#318 31 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In many timeseries workloads, the need to join one record to another based on recency or ordering is quite prevalent. This can be with two sensors that maintain different sampling rates, in handling market data (such as situations where you want to find the most recent quote for a given trade), or any other scenario where an ordering of events can be applied. In these situations, it is quite frequent that one set of data is at a *much* larger scale than the other, typically starting at an order of magnitude. It is also quite common that you would want to actually store your data *in* the appropriate order so as to minimize the effort when performing this kind of join operation (though that is not strictly the problem itself).

**Describe the solution you'd like**
Essentially the solution implemented by [Clickhouse ](https://clickhouse.tech/docs/en/sql-reference/statements/select/join/#asof-join-usage) and [KDB](https://code.kx.com/q/ref/aj/) (please note that the Clickhouse solution allows for the full breadth of ordering conditions for closest match).

**Describe alternatives you've considered**
There is a general [subquery solution](https://dba.stackexchange.com/questions/116424/left-join-on-closest-date) that can be used to achieve the desired outcome, but it is typically not performant and can be fairly awkward to express. Provided implementation of #141 and sufficient query optimization of the subquery solution into sort-merge join, it may not be necessary as direct syntax.

**Additional context**
- This likely depends on #141
- Related to https://github.com/apache/datafusion/issues/8393

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the dependency on #141 and the related issue #8393, then compare the requested behavior with the linked ClickHouse and KDB references. Done means DataFusion can express ASOF or specialized range joins for ordered, unevenly sampled data, including the intended ordering conditions and acceptable alternatives such as an optimized subquery.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.