Top down analysis
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 5
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I am trying (for the first time) to run a top down analysis using Quicksilver and [topdown.sh](https://github.com/LLNL/caliper-examples/blob/master/scripts/topdown/topdown.sh)
My json files (reported by Caliper/2.3.0) look a bit different
```
==> topdown_counters.json.GROUPBY_all <==
[
{"sum#libpfm.counter.BR_MISP_RETIRED.ALL_BRANCHES":"25482765","sum#libpfm.counter.CPU_CLK_UNHALTED.THREAD_P":"15515962999",...
==> topdown_counters.json.GROUPBY_function <==
[
{"sum#libpfm.counter.BR_MISP_RETIRED.ALL_BRANCHES":"17257","sum#libpfm.counter.CPU_CLK_UNHALTED.THREAD_P":"14372485",...
==> topdown_counters.json.GROUPBY_lulesh.phase <==
[
{"sum#libpfm.counter.BR_MISP_RETIRED.ALL_BRANCHES":"25586420",...
```
when compared with [quicksilver_topdown_counters.json](https://github.com/LLNL/caliper-examples/blob/master/datasets/quicksilver_topdown_counters.json.tar.bz2) (Caliper/1.7)
```
{"cali.event.set":"305","event.set#pthread.id":"46913237669632","cali.caliper.version":"1.7.0-dev","mpi.size":"16","lib
pfm.counter.BR_MISP_RETIRED:ALL_BRANCHES":"33","libpfm.counter.CPU_CLK_UNHALTED:THREAD_P":"86425","...
```
My configuration file is [0.conf.txt](https://github.com/LLNL/caliper-examples/files/4462213/0.conf.txt)
My problem is that i can not use the [jupyter](https://github.com/LLNL/caliper-examples/blob/master/jupyter_notebooks/Quicksilver%20Topdown%20Analysis.ipynb) notebook as it will fail with:
```
df = pd.concat(pd.read_json('./topdown_counters.json', orient='records', lines=True, chunksize=1000))
Traceback (most recent call last):
File "", line 1, in
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/core/reshape/concat.py", line 225, in concat
copy=copy, sort=sort)
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/core/reshape/concat.py", line 256, in __init__
objs = list(objs)
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/io/common.py", line 74, in
BaseIterator.next = lambda self: self.__next__()
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/io/json/json.py", line 570, in __next__
obj = self._get_object_parser(lines_json)
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/io/json/json.py", line 546, in _get_object_parser
obj = FrameParser(json, **kwargs).parse()
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/io/json/json.py", line 638, in parse
self._parse_no_numpy()
File "/apps/dom/UES/jenkins/7.0.UP01/gpu/easybuild/software/PyExtensions/2.7.15.7-CrayGNU-19.10/lib/python2.7/site-packages/pandas-0.23.3-py2.7-linux-x86_64.egg/pandas/io/json/json.py", line 867, in _parse_no_numpy
loads(json, precise_float=self.precise_float), dtype=None)
ValueError: Expected object or value
```
Did i miss something obvious ?
Thanks, jg.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with scripts/topdown/topdown.sh and jupyter_notebooks/Quicksilver Topdown Analysis.ipynb, then compare their expected JSON format with the Caliper 2.3.0 output and 0.conf.txt. Reproduce the pandas read_json failure and determine the required compatibility change; done means the notebook can load the generated data and perform the top-down analysis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, pandas, python, shell
- Domain
- data, performance, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100