feathr-ai / feathr-ai/feathr

UDF E2E tests failed

Open
#167 0 comments 1 reaction 0 assignees Claimed by @windoze View on GitHub
Dominant language
Scala
Stars
1.9k
Forks
247
PR merge metrics
No merged PRs in 30d

Description

After merging #139, UDF E2E tests failed as the PR changed to use `pyspark.cloudpickle` instead of `inspect.getsource()`.

The root cause is that the `cloudpickle` by default will **not** pickle modules by value, and the PyTest doesn't run test code in `__main__` as normal python execution flow, that means all UDFs defined in the test code belongs to a different module than `__main__`, this leads to a mysterious `ModuleNotFound` exception when pickled UDFs were used in the remote PySpark job, as it cannot find the module where the pickled function belongs to, because the test module doesn't exist on the remote side.

Since version 2.0, `cloudpickle` resolves this issue by involving a new `register_pickle_by_value(module)` function, but unfortunately this function doesn't exist in the `cloudpickle` bundled with PySpark, which stays at 1.6 for years.

This issue actually will **not** happen in typical usage, e.g. in a Python REPL env or Jupyter notebook, as they all run current code in `__main__` module, it only affects PyTest.

One possible solution is to use the new `cloudpickle` but it introduces new dependencies which must be installed on the remote Spark cluster manually.

The other solution is to let user package their UDF into a separated Python file and submit this file along with the Spark job, but this is not compatible with current test code thus needs a major restructuring.

The workaround is pretty straightforward, changing the module name to `__main__` at the very beginning of the E2E test script, it looks like a hack but actually does the job pretty well.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.