feature: basic results analysis for BehaviorSpace

Open
#489 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale

Research direction

The issue names no files, tests, or entry points; begin by locating BehaviorSpace's existing results and CSV handling. Narrow the broad notes to one agreed first capability, then verify it against a simple one-parameter experiment and the requested plot or export behavior.

Written by the indexing model from the issue text.

Description

behaviorspace

some old notes from a 2009 devel meeting on this:

General agreement, I think that we need to make sure that simple,
broad, basic low-threshold functionality is there before we do anything
too extensive or advanced in any particular area (within this general
area of results collection and analysis). currently it's quite a pain
to do even the simplest stuff, e.g. familiar examples like varying
the density in Fire and getting a plot of the resulting burn percentages

General agreement, I think, that the right next step is to do what
Forrest proposed, namely build something like Ben Shargel's original
tool, but instead of focusing on "heat maps" (which show two parameters
varying), focus on regular plots (which show one parameter varying).
such a UI can support one parameter experiments directly, and we should
support Paulo's proposed simplified output format. The same UI can
support multiple-parameter experiments as long as we require that you
pick one parameter at a time and fix the others temporarily in order to
get a plot.

Further detail/comments on Forrest's suggestion:

  • We should support multiple runs per combination of settings -- this is
    basic. We could just take the mean of the runs and show that, probably
    with an error bar or a mustache box or whatever it's called, or maybe
    just plot a dot for each run and let the user eyeball the dots.
  • Once you have a plot there should always be an export button to export
    just the data you are seeing Paulo-style.
  • It would be awesome if you could start examining the results as they
    come in, rather than having to wait until the entire experiment is over.
    We should probably plan for this from the start.
  • You will need be able to open a previously saved csv file and browse
    it using this UI.
  • It would be awesome, and maybe not that hard, if you could open
    multiple previously saved csv files and browse the combined results from
    all of them. That would save cluster people the trouble of merging their
    results files themselves.

Other notes:

  • Josh U recommended we look at TinkerPlots ("it kind of blew my mind
    for ease of use")
  • Michelle says working in this space would be easier if she/we didn't
    have to parse csv files and suck data out of plots. some kind of objects
    holding the results or events notifying of results as they happen (in
    NetLogoListener or whatever)
  • Seth said maybe "metrics" should be something that we
    explicitly represent so they can be shared by plots, monitors,
    behaviorspace, etc? currently the solution is to make a global variable
    and/or reporter procedure. but then you have to put that in the
    procedures tab. is that good or bad? this overlaps with the Two Joshes
    plotting work. maybe it's fine the way it is. or maybe some things
    could be made more convenient without making metrics separate and
    explicit, for example, when making an experiment setup you could choose
    from a list of reporters taken from monitors and plots.
Dominant language
Scala
Stars
1.2k
Forks
272
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NetLogo/NetLogo

All issues in NetLogo/NetLogo

Similar issues

More Scala issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.