feathr-ai / feathr-ai/feathr

[FR]Expose incremental feature generation in Python APIs.

Đang mở
#650 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
feature
Ngôn ngữ chính
Scala
Star
1.9k
Fork
247
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

### Willingness to contribute

No. I cannot contribute a bug fix at this time.

### Feature Request Proposal

This is a feature request from a few customers. Basically when generating features for sliding windowed joins, we need to have some way to cache the result (for example total_login_time_in_last_90_days, which can leverage the historical login in last 30 days and save some compute).

Sample config as below. Note that we need to put outputFormat as RAW_DATA.

Also note that this needs to be applied separately on different groupBy keys (e.g. 'viewer_actor' and 'viewer' in the code below).

operational: {

name: eAffinity2FeatureGen

endTime: NOW

endTimeFormat: "yyyy-MM-dd"

resolution: DAILY

enableIncremental = true

output: [

{

name: HDFS

outputFormat: RAW_DATA

params: {

path: "/jobs/affinity/eAffinity/eAffinity_2/aggFeatures/viewerAffinity"

num-parts: 20

features: [

viewer_impressionCount_365d,

viewer_clickCount_365d,

]

}

}

]

}

features: [

viewer_impressionCount_365d,

viewer_clickCount_365d,

]

This item mainly need to add this in the Python API and prepare the right test cases. Docs should be simple.

### Motivation

The motivation is to make it easier to reuse the pre-calculated features.

### Details

_No response_

### What component(s) does this feature request affect?

- [X] `Python Client`: This is the client users use to interact with most of our API. Mostly written in Python.
- [X] `Computation Engine`: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
- [ ] `Feature Registry API`: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
- [ ] `Feature Registry Web UI`: The Web UI for feature registry. Written in React

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.