feathr-ai / feathr-ai/feathr

[FR]Expose incremental feature generation in Python APIs.

Open
#650 0 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Scala
Stars
1.9k
Forks
247
PR merge metrics
No merged PRs in 30d

Description

### Willingness to contribute

No. I cannot contribute a bug fix at this time.

### Feature Request Proposal

This is a feature request from a few customers. Basically when generating features for sliding windowed joins, we need to have some way to cache the result (for example total_login_time_in_last_90_days, which can leverage the historical login in last 30 days and save some compute).

Sample config as below. Note that we need to put outputFormat as RAW_DATA.

Also note that this needs to be applied separately on different groupBy keys (e.g. 'viewer_actor' and 'viewer' in the code below).

operational: {

name: eAffinity2FeatureGen

endTime: NOW

endTimeFormat: "yyyy-MM-dd"

resolution: DAILY

enableIncremental = true

output: [

{

name: HDFS

outputFormat: RAW_DATA

params: {

path: "/jobs/affinity/eAffinity/eAffinity_2/aggFeatures/viewerAffinity"

num-parts: 20

features: [

viewer_impressionCount_365d,

viewer_clickCount_365d,

]

}

}

]

}

features: [

viewer_impressionCount_365d,

viewer_clickCount_365d,

]

This item mainly need to add this in the Python API and prepare the right test cases. Docs should be simple.

### Motivation

The motivation is to make it easier to reuse the pre-calculated features.

### Details

_No response_

### What component(s) does this feature request affect?

- [X] `Python Client`: This is the client users use to interact with most of our API. Mostly written in Python.
- [X] `Computation Engine`: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
- [ ] `Feature Registry API`: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
- [ ] `Feature Registry Web UI`: The Web UI for feature registry. Written in React

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.