Lightning-AI / Lightning-AI/pytorch-lightning
`NeptuneCallback` produces lots of `X-coordinates (step) must be strictly increasing` errors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
When Optuna is run in parallel mode (`n_jobs=-1`), with `NeptuneCallback`, I get:
`[neptune] [error ] Error occurred during asynchronous operation processing: X-coordinates (step) must be strictly increasing for series attribute: trials/values. Invalid point: 0.0`
It's normal that during parallel or distributed hyperparam optimization, information become unordered. Either Neptune should support adding steps out of order, or `NeptuneCallback` should support it somehow (e.g. by using an artificial step number).
### What version are you seeing the problem on?
v1.x
### How to reproduce the bug
```python
study.optimize(..., callbacks=[NeptuneCallback(run)], n_jobs=-1)
```
### Error messages and logs
`[neptune] [error ] Error occurred during asynchronous operation processing: X-coordinates (step) must be strictly increasing for series attribute: trials/values. Invalid point: 0.0`
### Environment
Any multi-threaded environment.
### More info
_No response_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the NeptuneCallback entry point and reproduce the issue with study.optimize(..., callbacks=[NeptuneCallback(run)], n_jobs=-1). Check how parallel trials are recorded in the trials/values series; done means the multi-threaded example no longer produces strictly increasing step errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100