HumanSignal / HumanSignal/label-studio-ml-backend

ner ml backend while training [job_result.json doesn't exist] [label_studio_ml.model::_get_latest_job_result_from_workdir::438]

Open
#342 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
490
Avg merge
1d 14h
Merged PRs (30d)
3

Description

Whenever I click start training, nothing happens, here is the logs.

```
`2023-08-30 14:26:02 server | [2023-08-30 08:41:02,166] [ERROR] [label_studio_ml.exceptions::exception_f::53] Traceback (most recent call last):
2023-08-30 14:26:02 server | File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
2023-08-30 14:26:02 server | return f(*args, **kwargs)
2023-08-30 14:26:02 server | File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 70, in _train
2023-08-30 14:26:02 server | job = _manager.train(annotations, project, label_config, **params)
2023-08-30 14:26:02 server | File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 683, in train
2023-08-30 14:26:02 server | project, label_config, train_kwargs=kwargs, tasks=tasks)
2023-08-30 14:26:02 server | File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 638, in train_script_wrapper
2023-08-30 14:26:02 server | train_output = m.model.fit(data_stream, workdir, **train_kwargs)
2023-08-30 14:26:02 server | File "./ner.py", line 474, in fit
2023-08-30 14:26:02 server | raise ValueError('Specify "WORK_DIR" environmental variable to store model checkpoints.')
2023-08-30 14:26:02 server | ValueError: Specify "WORK_DIR" environmental variable to store model checkpoints.
2023-08-30 14:26:02 server |
2023-08-30 14:26:02 server | Traceback (most recent call last):
2023-08-30 14:26:02 server | File "/usr/local/lib/python3.7/site-packages/label_studio_ml/exceptions.py", line 39, in exception_f
2023-08-30 14:26:02 server | return f(*args, **kwargs)
2023-08-30 14:26:02 server | File "/usr/local/lib/python3.7/site-packages/label_studio_ml/api.py", line 70, in _train
2023-08-30 14:26:02 server | job = _manager.train(annotations, project, label_config, **params)
2023-08-30 14:26:02 server | File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 683, in train
2023-08-30 14:26:02 server | project, label_config, train_kwargs=kwargs, tasks=tasks)
2023-08-30 14:26:02 server | File "/usr/local/lib/python3.7/site-packages/label_studio_ml/model.py", line 638, in train_script_wrapper
2023-08-30 14:26:02 server | train_output = m.model.fit(data_stream, workdir, **train_kwargs)
2023-08-30 14:26:02 server | File "./ner.py", line 474, in fit
2023-08-30 14:26:02 server | raise ValueError('Specify "WORK_DIR" environmental variable to store model checkpoints.')
2023-08-30 14:26:02 server | ValueError: Specify "WORK_DIR" environmental variable to store model checkpoints.
2023-08-30 14:26:02 server |
2023-08-30 14:26:02 server | [pid: 21|app: 0|req: 11/11] 172.18.0.1 () {36 vars in 739 bytes} [Wed Aug 30 08:41:02 2023] POST /train => generated 1048 bytes in 48 msecs (HTTP/1.1 500) 2 headers in 92 bytes (1 switches on core 0)
2023-08-30 14:26:02 server | [pid: 21|app: 0|req: 12/12] 172.18.0.1 () {32 vars in 672 bytes} [Wed Aug 30 08:41:02 2023] GET /health => generated 54 bytes in 1 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)
2023-08-30 14:26:02 server | [2023-08-30 08:41:02,378] [ERROR] [label_studio_ml.model::_get_latest_job_result_from_workdir::438] The latest job result file /data/models/43.1693383892/1693384862/job_result.json doesn't exist
2023-08-30 14:26:02 server | [2023-08-30 08:41:02,382] [ERROR] [label_studio_ml.model::_get_latest_job_result_from_workdir::438] The latest job result file /data/models/43.1693383892/1693384618/job_result.json doesn't exist
2023-08-30 14:26:02 server | [2023-08-30 08:41:02,386] [ERROR] [label_studio_ml.model::_get_latest_job_result_from_workdir::438] The latest job result file /data/models/43.1693383892/1693384588/job_result.json doesn't exist
2023-08-30 14:26:02 server | [2023-08-30 08:41:02,389] [ERROR] [label_studio_ml.model::_get_latest_job_result_from_workdir::438] The latest job result file /data/models/43.1693383892/1693384048/job_result.json doesn't exist
2023-08-30 14:26:02 server | [2023-08-30 08:41:02,392] [ERROR] [label_studio_ml.model::_get_latest_job_result_from_workdir::438] The latest job result file /data/models/43.1693383892/1693383970/job_result.json doesn't exist
2023-08-30 14:26:02 server | [2023-08-30 08:41:02,395] [ERROR] [label_studio_ml.model::_get_latest_job_result_from_workdir::438] The latest job result file /data/models/43.1693383892/1693383965/job_result.json doesn't exist
2023-08-30 14:26:02 server | [pid: 21|app: 0|req: 13/13] 172.18.0.1 () {36 vars in 724 bytes} [Wed Aug 30 08:41:02 2023] POST /setup => generated 31 bytes in 29 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)`
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.