baidu / baidu/DDParser

批量使用ddp.parse()方法报错

Open
#56 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
999
Forks
156
PR merge metrics
No merged PRs in 30d

Description

开发者你好,感谢你们的工作,这个工具很好用。
我在使用中出现这样一个问题,实例化ddparser之后,重复调用.parse方法,大概12万次以后,就会报一个 `RuntimeError: thread constructor failed: Resource temporarily unavailable`错误,我的代码及错误详情如下:

运行代码
```
from ddparser import DDParser

ddp = DDParser(prob=True, use_pos=True, buckets=True, use_cuda=False)
# text.json 是一个 文本list 文件,每行是一个短句
with open('text.json', 'rb') as f:
data = json.load(f)
rets = []
for line in data:
ret = ddp.parse([line])
rets.append(ret)
```

错误详情
```
File "/Users/wangy/PycharmProjects/test/data.py", line 385, in get_ddparser_results
ret = ddp.parse([line])
File "/Users/wangy/anaconda3/lib/python3.7/site-packages/ddparser/run.py", line 399, in parse
pred_arcs, pred_rels, pred_probs = epoch_predict(self.env, self.args, self.model, dataset.loader)
File "", line 2, in epoch_predict
File "/Users/wangy/anaconda3/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 276, in __impl__
return func(*args, **kwargs)
File "/Users/wangy/anaconda3/lib/python3.7/site-packages/ddparser/parser/model.py", line 185, in epoch_predict
for batch, inputs in enumerate(loader(), start=1):
File "/Users/wangy/anaconda3/lib/python3.7/site-packages/paddle/fluid/reader.py", line 960, in __iter__
self._init_iterable()
File "/Users/wangy/anaconda3/lib/python3.7/site-packages/paddle/fluid/reader.py", line 909, in _init_iterable
self._pin_memory)
RuntimeError: thread constructor failed: Resource temporarily unavailable
```
这里的 ddparser==1.0.6,paddlepaddle==2.1.3

感谢!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure by repeatedly calling DDParser.parse([line]) with ddparser 1.0.6 and PaddlePaddle 2.1.3. Start in ddparser/run.py at parse(), then follow parser/model.py's epoch_predict() into Paddle's reader initialization shown in the traceback. Done means large batches of repeated parsing no longer exhaust available thread resources or raise the reported RuntimeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.