dbt integration: FileSource generates invalid hardcoded paths
- Dominant language
- Python
- Stars
- 7.3k
- Forks
- 1.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 15
Description
## Context
PR #5827 added support for FileSource data sources in dbt integration.
## Problem
The FileSource implementation generates hardcoded paths that likely don't exist:
```python
# In mapper.py:240-242
return FileSource(
name=f"{model.name}_source",
path=f"/data/{model.name}.parquet", # Hardcoded, likely invalid
# ...
)
```
This path `/data/{model.name}.parquet` is unlikely to match actual file locations.
## Impact
- Generated FileSource objects won't work without manual editing
- Confusing user experience
- No clear documentation on how to fix the paths
## Proposed Solutions
### Option 1: Make path configurable
```bash
feast dbt import -m manifest.json -e driver_id --data-source-type file --file-path-template "/my/data/{model}.parquet"
```
### Option 2: Read from dbt meta
Support custom path in dbt model meta:
```yaml
# dbt model
models:
- name: driver_stats
meta:
feast:
file_path: "/data/warehouse/driver_stats.parquet"
```
### Option 3: Require explicit path mapping file
```bash
feast dbt import -m manifest.json -e driver_id --file-path-config paths.yaml
```
## Recommendation
Implement Option 1 (path template) as it's simplest and covers most use cases. Consider Option 2 for advanced scenarios.
## Related
- PR #5827
Contributor guide
Assessment
This issue has not been assessed yet.