esa / esa/pygmo2

speeding up ipyparallel evolutions [FEATURE]

Open
#156 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
536
Forks
74
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Sending the problem and population to every ipyparallel engine with every evaluation is fairly slow, specially when using many (>100) engines. If the problem does not change during the evolution, it should be enough to send it only once. Additionally, it may be worth having the option to read/write the population to disk, completely avoiding the scatter step.

**Describe the solution you'd like**
I think I'm looking for a new bfe, that does not pickle everything every time, and that can use other ipyparallel views (such as [BroadcastView](https://ipyparallel.readthedocs.io/en/latest/examples/broadcast/Broadcast%20view.html)). Would other aspects of pygmo have to change to enable this?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the existing ipyparallel bfe path and how it sends the problem and population to each engine. Compare that flow with the linked BroadcastView documentation and determine the scope of a bfe that avoids repeated pickling and scattering; done should include a defined approach for optional population disk I/O.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.