Issues running allentune on allennlp > 1.0.0
- 主要言語
- Python
- スター
- 141
- フォーク
- 13
- PR マージ指標
- PR 指標を取得中
説明
Hi! I realize that allentune is pinned to allennlp==1.0.0, but I was hoping to use it for some experiments on allennlp 1.3.0. I ran into the following issue that doesn't seem related to allennlp, and I was wondering if you had any pointers on how to fix it. Specifically, it looks like ray is trying to load allentune from GCS (something remote?) and having trouble unpickling the `allentune.commands` module, even though it's installed in my python environment. Do you have any pointers on workarounds or fixes?
Additional info: I was able to run `pytest -v .` on my installation, and all tests passed.
Here's how I'm invoking allentune:
```
i=ner_mt_mbert; allentune search --experiment-name $i --num-cpus 14 --log-dir ./tmp --search-space config/lib/search.json --num-samples 1 --base-config config/${i}.jsonnet --include-package modules
```
And the log trace:
```
2021-01-11 13:12:36,121 - INFO - allentune.modules.ray_executor - Init Ray with 14 CPUs and 1 GPUs.
2021-01-11 13:12:36,124 INFO resource_spec.py:212 -- Starting Ray with 104.54 GiB memory available for workers and up to 48.8 GiB for objects. You can adjust these settings with ray.init(memory=, object_store_memory=).
2021-01-11 13:12:36,518 WARNING services.py:923 -- Redis failed to start, retrying now.
2021-01-11 13:12:37,210 INFO services.py:1165 -- View the Ray dashboard at localhost:8265
2021-01-11 13:12:37,631 - INFO - allentune.modules.ray_executor - Run Configuration: {'ner_mt_mbert': {'run': 'run', 'resources_per_trial': {'cpu': 1, 'gpu': 1}, 'config': {'RANDOM_SEED': . at 0x7f0b1e6e17a0>, 'NUMPY_SEED': . at 0x7f0b1e6e97a0>, 'PYTORCH_SEED': . at 0x7f0b1e6e9440>}, 'local_dir': './tmp', 'num_samples': 1}}
2021-01-11 13:12:37,667 WARNING tune.py:318 -- Tune detects GPUs, but no trials are using GPUs. To enable trials to use GPUs, set tune.run(resources_per_trial={'gpu': 1}...) which allows Tune to expose 1 GPU to each trial. You can also override `Trainable.default_resource_request` if using the Trainable API.
== Status ==
Memory usage on this node: 156.3/251.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 1/14 CPUs, 1/1 GPUs, 0.0/104.54 GiB heap, 0.0/33.64 GiB objects
Result logdir: /homes/gws/echau18/research-lr-ssmba/tmp/ner_mt_mbert
Number of trials: 1 (1 RUNNING)
+-----------------+----------+-------+
| Trial name | status | loc |
|-----------------+----------+-------|
| run_bb506_00000 | RUNNING | |
+-----------------+----------+-------+
2021-01-11 13:12:38,520 WARNING worker.py:1047 -- Failed to unpickle actor class 'ImplicitFunc' for actor ID 45b95b1c0100. Traceback:
Traceback (most recent call last):
File "/homes/gws/echau18/miniconda3/envs/lr-ssmba/lib/python3.7/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs
actor_class = pickle.loads(pickled_class)
ModuleNotFoundError: No module named 'allentune.commands'
2021-01-11 13:12:38,525 ERROR trial_runner.py:520 -- Trial run_bb506_00000: Error processing event.
Traceback (most recent call last):
File "/homes/gws/echau18/miniconda3/envs/lr-ssmba/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 468, in _process_trial
result = self.trial_executor.fetch_result(trial)
File "/homes/gws/echau18/miniconda3/envs/lr-ssmba/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result
result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT)
File "/homes/gws/echau18/miniconda3/envs/lr-ssmba/lib/python3.7/site-packages/ray/worker.py", line 1474, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RuntimeError): ray::TemporaryActor.train() (pid=45844, ip=128.208.3.44)
File "python/ray/_raylet.pyx", line 407, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 442, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 445, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 446, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 400, in ray._raylet.execute_task.function_executor
RuntimeError: The actor with name ImplicitFunc failed to be imported, and so cannot execute this method.
(pid=45844) 2021-01-11 13:12:38,515 ERROR function_manager.py:496 -- Failed to load actor class ImplicitFunc.
(pid=45844) Traceback (most recent call last):
(pid=45844) File "/homes/gws/echau18/miniconda3/envs/lr-ssmba/lib/python3.7/site-packages/ray/function_manager.py", line 494, in _load_actor_class_from_gcs
(pid=45844) actor_class = pickle.loads(pickled_class)
(pid=45844) ModuleNotFoundError: No module named 'allentune.commands'
== Status ==
Memory usage on this node: 156.6/251.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/14 CPUs, 0/1 GPUs, 0.0/104.54 GiB heap, 0.0/33.64 GiB objects
Result logdir: /homes/gws/echau18/research-lr-ssmba/tmp/ner_mt_mbert
Number of trials: 1 (1 ERROR)
+-----------------+----------+-------+
| Trial name | status | loc |
|-----------------+----------+-------|
| run_bb506_00000 | ERROR | |
+-----------------+----------+-------+
Number of errored trials: 1
+-----------------+--------------+---------------------------------------------------------------------------------------------------+
| Trial name | # failures | error file |
|-----------------+--------------+---------------------------------------------------------------------------------------------------|
| run_bb506_00000 | 1 | /homes/gws/echau18/research-lr-ssmba/tmp/ner_mt_mbert/run_0_2021-01-11_13-12-37nbrmel3w/error.txt |
+-----------------+--------------+---------------------------------------------------------------------------------------------------+
2021-01-11 13:12:38,550 - ERROR - allentune.modules.ray_executor - Error during run of experiment 'ner_mt_mbert': ('Trials did not complete', [run_bb506_00000])
```
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。