llnl / llnl/caliper-examples

Top down analysis

Open
#5 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
5
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Hi,

I am trying (for the first time) to run a top down analysis using Quicksilver and [topdown.sh](https://github.com/LLNL/caliper-examples/blob/master/scripts/topdown/topdown.sh)
My json files (reported by Caliper/2.3.0) look a bit different
```
==> topdown_counters.json.GROUPBY_all <==
[
{"sum#libpfm.counter.BR_MISP_RETIRED.ALL_BRANCHES":"25482765","sum#libpfm.counter.CPU_CLK_UNHALTED.THREAD_P":"15515962999",...

==> topdown_counters.json.GROUPBY_function <==
[
{"sum#libpfm.counter.BR_MISP_RETIRED.ALL_BRANCHES":"17257","sum#libpfm.counter.CPU_CLK_UNHALTED.THREAD_P":"14372485",...
==> topdown_counters.json.GROUPBY_lulesh.phase <==
[
{"sum#libpfm.counter.BR_MISP_RETIRED.ALL_BRANCHES":"25586420",...
```

when compared with [quicksilver_topdown_counters.json](https://github.com/LLNL/caliper-examples/blob/master/datasets/quicksilver_topdown_counters.json.tar.bz2) (Caliper/1.7)

```
{"cali.event.set":"305","event.set#pthread.id":"46913237669632","cali.caliper.version":"1.7.0-dev","mpi.size":"16","lib
pfm.counter.BR_MISP_RETIRED:ALL_BRANCHES":"33","libpfm.counter.CPU_CLK_UNHALTED:THREAD_P":"86425","...
```

My configuration file is [0.conf.txt](https://github.com/LLNL/caliper-examples/files/4462213/0.conf.txt)

My problem is that i can not use the [jupyter](https://github.com/LLNL/caliper-examples/blob/master/jupyter_notebooks/Quicksilver%20Topdown%20Analysis.ipynb) notebook as it will fail with:

```
df = pd.concat(pd.read_json('./topdown_counters.json', orient='records', lines=True, chunksize=1000))

Traceback (most recent call last):
File "", line 1, in
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/core/reshape/concat.py", line 225, in concat
copy=copy, sort=sort)
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/core/reshape/concat.py", line 256, in __init__
objs = list(objs)
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/io/common.py", line 74, in
BaseIterator.next = lambda self: self.__next__()
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/io/json/json.py", line 570, in __next__
obj = self._get_object_parser(lines_json)
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/io/json/json.py", line 546, in _get_object_parser
obj = FrameParser(json, **kwargs).parse()
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/io/json/json.py", line 638, in parse
self._parse_no_numpy()
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/io/json/json.py", line 867, in _parse_no_numpy
loads(json, precise_float=self.precise_float), dtype=None)
ValueError: Expected object or value
```

Did i miss something obvious ?

Thanks, jg.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/topdown/topdown.sh and jupyter_notebooks/Quicksilver Topdown Analysis.ipynb, then compare their expected JSON format with the Caliper 2.3.0 output and 0.conf.txt. Reproduce the pandas read_json failure and determine the required compatibility change; done means the notebook can load the generated data and perform the top-down analysis.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, pandas, python, shell
Domain
data, performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.