On-demand feature view UDFs lose their module globals when rebuilt from the registry, so serving fails with NameError
- Dominant language
- Python
- Stars
- 7.3k
- Forks
- 1.4k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 15
Description
## Expected Behavior
An OnDemandFeatureView whose UDF calls a helper (or reads a constant/import) defined
at its own module level should keep working when the view is loaded back from the
registry — e.g. in a feature server, or any process that did not define the view itself.
## Current Behavior
The UDF is rebuilt from its stored source text only, so every name it resolves from
its defining module is missing, and the transformation fails at call time:
```
NameError: name 'scaled' is not defined
```
`PandasTransformation.from_proto` / `PythonTransformation.from_proto` prefer
`rehydrate_udf_from_source` over the stored dill body. That function `exec`s **only the
function's own source** into a fresh namespace seeded with just `pd`/`pandas`/`np`/`numpy`
(`sdk/python/feast/transformation/udf_rehydrate.py:102`), so a module-level helper,
constant, or an import under any other alias is not present.
The `exec` itself still succeeds, because a function body's free variables are only
resolved when the function is *called*. So `resolve_udf`
(`sdk/python/feast/transformation/udf_rehydrate.py:160`) treats rehydration as
successful and **never falls back to the dill body** — even when that body would have
worked. The failure surfaces later, during retrieval.
## Steps to reproduce
Apply an ODFV from a script/notebook, so dill serializes the UDF by value and captures
its globals:
```python
SCALE = 100.0
def scaled(x):
return x * SCALE
@on_demand_feature_view(
sources=[stats, req],
schema=[Field(name="conv_rate_scaled", dtype=Float64)],
)
def scaled_rate(inputs: pd.DataFrame) -> pd.DataFrame:
df = pd.DataFrame()
df["conv_rate_scaled"] = scaled(inputs["conv_rate"]) + inputs["bonus"]
return df
fs.apply([driver, src, stats, req, scaled_rate])
fs.materialize_incremental(datetime(2026, 6, 1))
```
Then read it back in a separate process that only has the registry (what a feature
server does):
```python
fs = FeatureStore(repo_path=".")
fs.get_online_features(
features=["driver_stats:conv_rate", "scaled_rate:conv_rate_scaled"],
entity_rows=[{"driver_id": 1001, "bonus": 1.0}],
).to_dict()
# NameError: name 'scaled' is not defined
```
Monkeypatching `from_proto` back to the previous dill-only behaviour, in the same
process and against the same registry, returns the correct result:
```python
@classmethod
def _old_from_proto(cls, p):
return PandasTransformation(udf=dill.loads(p.body), udf_string=p.body_text)
PandasTransformation.from_proto = _old_from_proto
# -> conv_rate_scaled=[51.0]
```
So the registry demonstrably holds a working body that is being ignored.
Both transformation modes are affected — a bare proto round-trip is enough to show it:
```python
t = PythonTransformation(udf=py_udf, udf_string=src) # py_udf calls helper()
rt = PythonTransformation.from_proto(t.to_proto())
t.udf({"a": [1, 2]}) # {'out': [3, 6]}
rt.udf({"a": [1, 2]}) # NameError: name 'helper' is not defined
```
### Specifications
- Version: `master` @ `5ad5592390febfca60c9d88edf7daccbdd156fd6` (source-first rehydration added in #6655)
- Platform: Linux x86_64, Python 3.11.15
- Subsystem: on-demand feature views / transformation serialization
## Possible Solution
Make the "is this rehydrated callable usable?" check non-vacuous before accepting it.
Walk the compiled function's `co_names` (recursively through nested code objects) and
confirm each is resolvable in the exec namespace or in builtins; if any name is missing,
return `None` so `resolve_udf` falls back to the dill body as it did before.
A `NameError`-catching wrapper that retries via dill on first call would also work, but
validating up front keeps the failure out of the serving path entirely.
Worth noting this is distinct from #5620, which covers the dill-by-reference case where
the defining module genuinely is not importable. The problem here is that source-first
rehydration is chosen even when the stored dill body would succeed, so a setup that
previously served correctly now raises.
I'm happy to put up a PR for this if the validate-then-fall-back approach looks right.
Contributor guide
Research direction
Start with sdk/python/feast/transformation/udf_rehydrate.py, especially rehydrate_udf_from_source and resolve_udf, then inspect PandasTransformation.from_proto and PythonTransformation.from_proto. Reproduce the bare proto round-trip with a helper-based UDF; done means module globals no longer cause serving-time NameError and a usable stored dill body is used when source rehydration is incomplete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100