speeding up ipyparallel evolutions [FEATURE]
- Dominant language
- C++
- Stars
- 536
- Forks
- 74
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Sending the problem and population to every ipyparallel engine with every evaluation is fairly slow, specially when using many (>100) engines. If the problem does not change during the evolution, it should be enough to send it only once. Additionally, it may be worth having the option to read/write the population to disk, completely avoiding the scatter step.
**Describe the solution you'd like**
I think I'm looking for a new bfe, that does not pickle everything every time, and that can use other ipyparallel views (such as [BroadcastView](https://ipyparallel.readthedocs.io/en/latest/examples/broadcast/Broadcast%20view.html)). Would other aspects of pygmo have to change to enable this?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the existing ipyparallel bfe path and how it sends the problem and population to each engine. Compare that flow with the linked BroadcastView documentation and determine the scope of a bfe that avoids repeated pickling and scattering; done should include a defined approach for optional population disk I/O.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100