NSLS2 / NSLS2/pyCHX

extracting onekey from pyCHX results file returns an empty dictionary

Open
#68 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug discussion V1 V2
Dominant language
Jupyter Notebook
Stars
8
Forks
12
Avg merge
2h 53m
Merged PRs (30d)
6

Description

https://github.com/NSLS-II/pyCHX/blob/6ae02935814caa0ca1efe413bc480a74c4c8fb94/pyCHX/Create_Report.py#L1826
Using this function with default arguments to extract data. Works well, but so much data takes a long time. I tried to speed up by only asking for what I want. I tried the follow:

  • onekey argument
  • excluded_keys argument

excluded_keys

this is the best hack around accessing just 1 key so you are not pulling all 2-time and roi intensity with time data if you just want an average image or g2b and tb. Did notice, that accessing 2time data (g12b ) is only 2 seconds on the server, but the gateway must be bogging down the communication to the notebook.

onekey

tried onekey = 'md', onekey = 'g12b', onekey = 'avg_img', onekey = 'imgsum' and all behaved the same way.

It seems that https://github.com/NSLS-II/pyCHX/blob/6ae02935814caa0ca1efe413bc480a74c4c8fb94/pyCHX/v2/_futurepyCHX/Create_Report.py#L2192 tried to implement the approach i used in my own editing of the function.

For my own function, the change i made doesn't seem to help though it seems to me to be exactly what was done in the debug function for the 'onekey' not None block. The debug function cannot extract md or g12b. Didn't try others.

For the the "_debug" version, not applying any extra arguments, there is an error so it seems this is also a work in progress. This is with default excluded_keys and default onekey

----> 2 total_res = load_data_from_pipeline_onekey(uid,)# onekey_s = 'md')#onekey='g12b')

/XF11ID/analysis/2018_3/lwiegart/development/Auto_Analysis_Functions_dev1.py in load_data_from_pipeline_onekey(uid, onekey_s)
    29                 #print('also tried: '+inDir+'uid=%s_Res.h5'%uid_full)
    30                 input_filename='uid=%s_Res.h5'%uid_full
---> 31                 total_res[u] = extract_xpcs_results_from_h5_debug( filename = 'uid=%s__Res.h5'%uid_full, import_dir = inDir,exclude_keys=exclude_keys, onekey = onekey_s )
    32 
    33         #total_res[uid] = extract_xpcs_results_from_h5( filename = 'uid=%s_phi_Res.h5'%uid_full, import_dir = inDir+ uid + '/' )

/XF11ID/analysis/2018_3/lwiegart/development/Auto_Analysis_Functions_dev1.py in extract_xpcs_results_from_h5_debug(filename, import_dir, onekey, exclude_keys)
    29                 if key not in exclude_keys:
    30                     if key in dicts:
---> 31                         extract_dict[key] = recursively_load_dict_contents_from_group(
    32                             hf, "/" + key + "/"
    33                         )

NameError: name 'recursively_load_dict_contents_from_group' is not defined

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pyCHX/Create_Report.py around line 1826 and compare pyCHX/v2/_futurepyCHX/Create_Report.py around line 2192, then reproduce the onekey calls for md and g12b. Trace the debug path and its missing recursively_load_dict_contents_from_group name. Done means onekey returns the requested data without an empty dictionary, and the default debug call no longer raises the reported NameError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.