feathr-ai / feathr-ai/feathr

[BUG] Synapse jobs fail with LIVY_JOB_STATE_DEAD

Abierto
#654 4 comentarios 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Scala
Estrellas
1.9k
Forks
247
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

### Willingness to contribute

No. I cannot contribute a bug fix at this time.

### Feathr version

v0.7.2

### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 20.0)**: Synapse Spark Pool
- **Python version**: 3.8
- **Spark version, if reporting runtime issue**: 3.2

### Describe the problem

Synapse Spark jobs fail to start and fail with `LIVY_JOB_STATE_DEAD`.

### Tracking information

```
Error details
Error details
This application failed due to the total number of errors: 1.
Error code 1
LIVY_JOB_STATE_DEAD

Message
[plugins.mrhrdeastus.pool1.18 WorkspaceType: CCID:<5f614438-8afc-4adc-b95c-bdb54f062c03>] [Monitoring] Livy Endpoint=[https://hubservice1.eastus.azuresynapse.net:8001/api/v1.0/publish/6fac8da0-7a3e-41c3-b993-750dfb8ab446]. Livy Id=[0] Job failed during run time with state=[dead].

Source
Unknown
```

### Code to reproduce bug

Run the `databricks_quickstart_nyc_taxi_driver` or `product_recommendation_demo` sample notebook with Synapse

The error occurs when waiting for the `client.get_offline_features` call to complete.

### What component(s) does this bug affect?

- [ ] `Python Client`: This is the client users use to interact with most of our API. Mostly written in Python.
- [X] `Computation Engine`: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
- [ ] `Feature Registry API`: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
- [ ] `Feature Registry Web UI`: The Web UI for feature registry. Written in React

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.