baidu / baidu/lac

python 多进程调用失败

Open
#264 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
4k
Forks
592
PR merge metrics
No merged PRs in 30d

Description

```python3
# SentenceAnalysis() ## 这里做实例化lac的模型
pool=Pool(processes=30)
res= pool.map(SentenceAnalysis().sentence_struct,querylist) ## sentence_struct就是用的lac来iqu词性、词的重要性等操作
```
Traceback (most recent call last):
File "/mnt/data/yzy/project/DataPrepare/bin/VoiceDemand.py", line 199, in
res= pool.map(sentenceanaly.sentence_struct,querylist)
File "/home/yiziyang/.conda/envs/nlu_train/lib/python3.7/multiprocessing/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/yiziyang/.conda/envs/nlu_train/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
File "/home/yiziyang/.conda/envs/nlu_train/lib/python3.7/multiprocessing/pool.py", line 431, in _handle_tasks
put(task)
File "/home/yiziyang/.conda/envs/nlu_train/lib/python3.7/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/home/yiziyang/.conda/envs/nlu_train/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: can't pickle paddle.fluid.core_avx.CPUPlace objects

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure from VoiceDemand.py line 199, focusing on multiprocessing.Pool.map with SentenceAnalysis().sentence_struct and the reported paddle.fluid.core_avx.CPUPlace pickling error. Trace how the LAC model is instantiated and determine a supported multiprocessing usage; done means the call processes querylist without the serialization error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.