feast-dev / feast-dev/feast

dbt integration: FileSource generates invalid hardcoded paths

Open
#5,874 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
1d 21h
Merged PRs (30d)
15

Description

## Context
PR #5827 added support for FileSource data sources in dbt integration.

## Problem
The FileSource implementation generates hardcoded paths that likely don't exist:

```python
# In mapper.py:240-242
return FileSource(
name=f"{model.name}_source",
path=f"/data/{model.name}.parquet", # Hardcoded, likely invalid
# ...
)
```

This path `/data/{model.name}.parquet` is unlikely to match actual file locations.

## Impact
- Generated FileSource objects won't work without manual editing
- Confusing user experience
- No clear documentation on how to fix the paths

## Proposed Solutions

### Option 1: Make path configurable
```bash
feast dbt import -m manifest.json -e driver_id --data-source-type file --file-path-template "/my/data/{model}.parquet"
```

### Option 2: Read from dbt meta
Support custom path in dbt model meta:
```yaml
# dbt model
models:
- name: driver_stats
meta:
feast:
file_path: "/data/warehouse/driver_stats.parquet"
```

### Option 3: Require explicit path mapping file
```bash
feast dbt import -m manifest.json -e driver_id --file-path-config paths.yaml
```

## Recommendation
Implement Option 1 (path template) as it's simplest and covers most use cases. Consider Option 2 for advanced scenarios.

## Related
- PR #5827

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.