feathr-ai / feathr-ai/feathr

[FR]Expose incremental feature generation in Python APIs.

Abierto
#650 0 comentarios 0 reacciones 0 asignados Ver en GitHub
feature
Lenguaje dominante
Scala
Estrellas
1.9k
Forks
247
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

### Willingness to contribute

No. I cannot contribute a bug fix at this time.

### Feature Request Proposal

This is a feature request from a few customers. Basically when generating features for sliding windowed joins, we need to have some way to cache the result (for example total_login_time_in_last_90_days, which can leverage the historical login in last 30 days and save some compute).

Sample config as below. Note that we need to put outputFormat as RAW_DATA.

Also note that this needs to be applied separately on different groupBy keys (e.g. 'viewer_actor' and 'viewer' in the code below).

operational: {

name: eAffinity2FeatureGen

endTime: NOW

endTimeFormat: "yyyy-MM-dd"

resolution: DAILY

enableIncremental = true

output: [

{

name: HDFS

outputFormat: RAW_DATA

params: {

path: "/jobs/affinity/eAffinity/eAffinity_2/aggFeatures/viewerAffinity"

num-parts: 20

features: [

viewer_impressionCount_365d,

viewer_clickCount_365d,

]

}

}

]

}

features: [

viewer_impressionCount_365d,

viewer_clickCount_365d,

]

This item mainly need to add this in the Python API and prepare the right test cases. Docs should be simple.

### Motivation

The motivation is to make it easier to reuse the pre-calculated features.

### Details

_No response_

### What component(s) does this feature request affect?

- [X] `Python Client`: This is the client users use to interact with most of our API. Mostly written in Python.
- [X] `Computation Engine`: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
- [ ] `Feature Registry API`: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
- [ ] `Feature Registry Web UI`: The Web UI for feature registry. Written in React

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.