simonsobs / simonsobs/sodetlib

Estimate phase delay (TimeoutError) failure running with the pysmurf controller

Open
#282 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5
Forks
0
Avg merge
3d 17h
Merged PRs (30d)
4

Description

Seems like there's more EPICS timeout errors with all smurfs running in parallel. We've seen this twice in a row when trying to estimate phase delay on one slot while the other 5 are running their setup procedures.

2022-09-06T14:59:11+0000 Traceback (most recent call last):
  File "/readout-script-dev/jlashner/penn-latr/enable_optical_amps.py", line 22, in <module>

2022-09-06T14:59:11+0000     S.estimate_phase_delay(b)
  File "/usr/local/src/pysmurf/python/pysmurf/client/util/pub.py", line 50, in wrapper

2022-09-06T14:59:11+0000     rv = func(S, *args, **kwargs)
  File "/usr/local/src/pysmurf/python/pysmurf/client/util/smurf_util.py", line 292, in estimate_phase_delay

2022-09-06T14:59:11+0000     save_data=save_data, n_scan=n_scan)
  File "/usr/local/src/pysmurf/python/pysmurf/client/util/pub.py", line 50, in wrapper

2022-09-06T14:59:11+0000     rv = func(S, *args, **kwargs)
  File "/usr/local/src/pysmurf/python/pysmurf/client/tune/smurf_tune.py", line 645, in full_band_resp

2022-09-06T14:59:11+0000     adc_is_saturated = self.check_adc_saturation(band)
  File "/usr/local/src/pysmurf/python/pysmurf/client/util/pub.py", line 50, in wrapper
    rv = func(S, *args, **kwargs)
  File "/usr/local/src/pysmurf/python/pysmurf/client/util/smurf_util.py", line 1489, in check_adc_saturation

2022-09-06T14:59:11+0000     save_data=False, show_plot=False, save_plot=False)
  File "/usr/local/src/pysmurf/python/pysmurf/client/util/pub.py", line 50, in wrapper

2022-09-06T14:59:11+0000     rv = func(S, *args, **kwargs)
  File "/usr/local/src/pysmurf/python/pysmurf/client/util/smurf_util.py", line 1575, in read_adc_data

2022-09-06T14:59:11+0000     hw_trigger=hw_trigger)
  File "/usr/local/src/pysmurf/python/pysmurf/client/util/pub.py", line 50, in wrapper

2022-09-06T14:59:11+0000     rv = func(S, *args, **kwargs)
  File "/usr/local/src/pysmurf/python/pysmurf/client/util/smurf_util.py", line 1464, in read_stream_data_daq

2022-09-06T14:59:11+0000     sg.wait()
  File "/usr/local/src/pysmurf/python/pysmurf/client/command/sync_group.py", line 69, in wait

2022-09-06T14:59:11+0000     raise Exception('Timeout waiting for PVs to update.')

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with enable_optical_amps.py and follow estimate_phase_delay through full_band_resp, check_adc_saturation, read_stream_data_daq, and sync_group.py. Reproduce phase-delay estimation while the other SMuRFs run setup procedures, then investigate the PV update timeout. Done means the estimation completes without the reported TimeoutError under parallel operation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.