TimelyDataflow / TimelyDataflow/timely-dataflow

Primes example... slower with more workers?

Open
#479 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.6k
Forks
293
Avg merge
14h 46m
Merged PRs (30d)
4

Description

Following along here:

https://timelydataflow.github.io/timely-dataflow/chapter_1/chapter_1_1.html

I've downloaded a fresh copy of timely and find when I run the "parallel primes" program, timely gets much slower with more workers, opposite of what I'd expect. Is there a recent regression with Timely?

``
$ time cargo run --release --example primes -- -w16 > output16.txt
Finished release [optimized + debuginfo] target(s) in 0.02s
Running target/release/examples/primes -w16

real 1m9.235s
user 15m27.958s
sys 0m1.987s

$ time cargo run --release --example primes -- -w1 > output1.txt
Finished release [optimized + debuginfo] target(s) in 0.02s
Running target/release/examples/primes -w1

real 0m0.279s
user 0m0.204s
sys 0m0.076s

``

#![allow(unused_variables)]
extern crate timely;

use timely::dataflow::{InputHandle};
use timely::dataflow::operators::{Input, Exchange, Inspect, Probe};

fn main() {
    // initializes and runs a timely dataflow.
    timely::execute_from_args(std::env::args(), |worker| {

        let index = worker.index();
        let mut input = InputHandle::new();

        // create a new input, exchange data, and inspect its output
        let probe = worker.dataflow(|scope| {
            scope.input_from(&mut input)
                 .exchange(|x| *x)
                 .inspect( //move |x| println!("worker {}:\thello {}", index, x))
                          |x| {
                              let limit = (*x as f64).sqrt() as u64;
                              if *x > 1 && (2 .. limit + 1).all(|i| x % i > 0) {
                                  // why can't i capture index?
                                  println!("{} is prime", x);
                              }
                          })
                 .probe();
        });

        // introduce data and watch!
        for round in 0..200000 {
            if index == 0 {
                input.send(round);
            }
            input.advance_to(round + 1);
        }
    }).unwrap();
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the parallel primes example from chapter_1/chapter_1_1.html and reproduce it using cargo run --release --example primes with -w1 and -w16. Compare the worker and elapsed times, then trace the timely execute_from_args, dataflow, exchange, and probe entry points. Done means identifying whether the slowdown is a regression and documenting or fixing its cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.