alibaba / alibaba/Elastic-Federated-Learning-Solution

请问efls-data的单机部署构建镜像时一定要求FLINK-K8S环境吗

Open
#17 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
143
Forks
41
PR merge metrics
No merged PRs in 30d

Description

https://github.com/alibaba/Elastic-Federated-Learning-Solution/blob/master/docs/English/Standalone_Deployment_CN.md
我参照上面这个单机部署教程后,运行`python /xfl/test/test_data_join.py`后报错如下
是由`env = StreamExecutionEnvironment.get_execution_environment()`这里引起的
请问一定要在k8s上部署flink环境再启动docker镜像才可以部署单机模式么?

```
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "Thread-4" java.lang.NoClassDefFoundError: org/apache/flink/table/functions/python/PythonFunction
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetPublicMethods(Class.java:2902)ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/py4j/java_gateway.py", line 1188, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/py4j/java_gateway.py", line 1014, in send_command
response = connection.send_command(command)
File "/usr/local/lib/python3.7/dist-packages/py4j/java_gateway.py", line 1193, in send_command
"Error while receiving", e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving

at java.lang.Class.getMethods(Class.java:1615)
at sun.misc.ProxyGenerator.generateClassFile(ProxyGenerator.java:451)
at sun.misc.ProxyGenerator.generateProxyClass(ProxyGenerator.java:339)
at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:639)
at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:557)
at java.lang.reflect.WeakCache$Factory.get(WeakCache.java:230)Traceback (most recent call last):

File "./test_data_join.py", line 180, in
at java.lang.reflect.WeakCache.get(WeakCache.java:127)
at java.lang.reflect.Proxy.getProxyClass0(Proxy.java:419)
at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:719)
at org.apache.flink.api.python.shaded.py4j.Gateway.createProxy(Gateway.java:368)
at org.apache.flink.api.python.shaded.py4j.Protocol.getPythonProxy(Protocol.java:433)
at org.apache.flink.api.python.shaded.py4j.Protocol.getObject(Protocol.java:311)
t.test_psi_join()
at org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.getArguments(AbstractCommand.java:82) File "./test_data_join.py", line 174, in test_psi_join

at org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:77)
at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:750) run_client_and_server()

File "./test_data_join.py", line 91, in run_client_and_server
Caused by: java.lang.ClassNotFoundException: org.apache.flink.table.functions.python.PythonFunction
'example_id', 'example_id', 8)
at java.net.URLClassLoader.findClass(URLClassLoader.java:387) File "./test_data_join2.py", line 78, in __init__

at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
conf=conf)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
File "/xfl/xfl/data/pipelines.py", line 71, in __init__
... 19 more
env = get_flink_batch_env(conf)
File "/xfl/xfl/data/pipelines.py", line 41, in get_flink_batch_env
env = StreamExecutionEnvironment.get_execution_environment()
File "/usr/local/lib/python3.7/dist-packages/pyflink/datastream/stream_execution_environment.py", line 688, in get_execution_environment
gateway = get_gateway()
File "/usr/local/lib/python3.7/dist-packages/pyflink/java_gateway.py", line 75, in get_gateway
_gateway.entry_point.put("PythonFunctionFactory", PythonFunctionFactory())
File "/usr/local/lib/python3.7/dist-packages/py4j/java_gateway.py", line 1286, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/local/lib/python3.7/dist-packages/pyflink/util/exceptions.py", line 146, in deco
return f(*a, **kw)
File "/usr/local/lib/python3.7/dist-packages/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling t.put
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with docs/English/Standalone_Deployment_CN.md and reproduce python /xfl/test/test_data_join.py. Inspect xfl/data/pipelines.py and the reported PyFlink class-loading error to determine the standalone deployment prerequisites. Done means the documented setup is accurate and test_data_join.py runs without the ClassNotFoundException.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, python
Domain
data-engineering, devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.