[FR]Expose incremental feature generation in Python APIs.
- Ngôn ngữ chính
- Scala
- Star
- 1.9k
- Fork
- 247
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
### Willingness to contribute
No. I cannot contribute a bug fix at this time.
### Feature Request Proposal
This is a feature request from a few customers. Basically when generating features for sliding windowed joins, we need to have some way to cache the result (for example total_login_time_in_last_90_days, which can leverage the historical login in last 30 days and save some compute).
Sample config as below. Note that we need to put outputFormat as RAW_DATA.
Also note that this needs to be applied separately on different groupBy keys (e.g. 'viewer_actor' and 'viewer' in the code below).
operational: {
name: eAffinity2FeatureGen
endTime: NOW
endTimeFormat: "yyyy-MM-dd"
resolution: DAILY
enableIncremental = true
output: [
{
name: HDFS
outputFormat: RAW_DATA
params: {
path: "/jobs/affinity/eAffinity/eAffinity_2/aggFeatures/viewerAffinity"
num-parts: 20
features: [
viewer_impressionCount_365d,
viewer_clickCount_365d,
]
}
}
]
}
features: [
viewer_impressionCount_365d,
viewer_clickCount_365d,
]
This item mainly need to add this in the Python API and prepare the right test cases. Docs should be simple.
### Motivation
The motivation is to make it easier to reuse the pre-calculated features.
### Details
_No response_
### What component(s) does this feature request affect?
- [X] `Python Client`: This is the client users use to interact with most of our API. Mostly written in Python.
- [X] `Computation Engine`: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
- [ ] `Feature Registry API`: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
- [ ] `Feature Registry Web UI`: The Web UI for feature registry. Written in React
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.