Not enough time after loading defaults before trying to query SmurfVersion

This is @jlashner posting from Sam's account.

Seems like when running setup, there is not enough time after setting defaults before trying to query the SmurfVersion. In the logs below, Smurf version needs to be queried 5 times before it returns a value, and this has recently caused the setup to crash entirely (as seen here: https://github.com/simonsobs/sodetlib/discussions/353#discussion-5290857). After setup fails, though I'm able to caget the SmurfVersion so I think it's just a matter of waiting longer, or setting a longer timeout for this one particular caget call.

[ 2023-06-14 15:51:40 ]  caput smurf_server_s2:AMCc:FpgaTopLevel:AppTop:AppCore:MicrowaveMuxCore[1]:DBG:dacReset[1] 0
[ 2023-06-14 15:51:40 ]  caput smurf_server_s2:AMCc:ReadAll 1
[ 2023-06-14 15:51:44 ]  Waiting 20.00 seconds after...
[ 2023-06-14 15:52:04 ]  Done waiting.
[ 2023-06-14 15:52:04 ]  caget smurf_server_s2:AMCc:SmurfApplication:SmurfVersion
[ 2023-06-14 15:52:04 ]  0+unknown
[ 2023-06-14 15:52:04 ]  caput smurf_server_s2:AMCc:setDefaults 1
[ 2023-06-14 15:52:34 ]  Waiting 30.00 seconds after...
CA.Client.Exception...............................................
    Warning: "Virtual circuit unresponsive"
    Context: "localhost:34607"
    Source File: ../tcpiiu.cpp line 919
    Current Time: Wed Jun 14 2023 15:52:39.569473338
..................................................................
[ 2023-06-14 15:53:04 ]  Done waiting.
[ 2023-06-14 15:53:04 ]  caget smurf_server_s2:AMCc:SmurfApplication:SmurfVersion
/usr/local/lib/python3.8/dist-packages/epics/ca.py:1528: UserWarning: ca.get('smurf_server_s2:AMCc:SmurfApplication:SmurfVersion') timed out after 1.95 seconds.
  warnings.warn(msg % (name(chid), timeout))
[ 2023-06-14 15:53:06 ]  Command failed: smurf_server_s2:AMCc:SmurfApplication:SmurfVersion
[ 2023-06-14 15:53:06 ]  Retry attempt 1 of 5
[ 2023-06-14 15:53:08 ]  Retry attempt 2 of 5
[ 2023-06-14 15:53:10 ]  Retry attempt 3 of 5
[ 2023-06-14 15:53:12 ]  Retry attempt 4 of 5
[ 2023-06-14 15:53:14 ]  Retry attempt 5 of 5
[ 2023-06-14 15:53:15 ]  0+unknown
[ 2023-06-14 15:53:15 ]  The `SmurfApplication:JesdStatus` register is not implemented for pysmurf core code versions <4.1.0 (current version is 0).

Start by locating the setup routine that issues setDefaults and then queries SmurfApplication:SmurfVersion. Reproduce the caget timeout and retry sequence described in the logs, then examine the wait or timeout handling for this query. Done means setup reliably obtains SmurfVersion after loading defaults without crashing.