DiamondLightSource / DiamondLightSource/fastcs
Add OpenTelemetry tracing of function calls from transport user input to controller IO call
- Dominant language
- Python
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 3
Description
Add `start_trace` and `stop_trace` methods to `Tracer`. Where `log_event` can be used to log trace level messages about a instantaneous point in time, these new methods can be used to trace the start and stop of some long process. These methods will start and stop spans and publish via [OpenTelemetry](https://opentelemetry.io/docs/languages/python/getting-started/) as well as publishing a log message (this could be optional so that spans can be published without logging).
For example, the two `log_event` calls in [`wait_for_predicate`](https://github.com/DiamondLightSource/FastCS/blob/c6ce4389f0cc5842d3f0e1b6327d40a69d896cdc/src/fastcs/attributes/attr_r.py#L148) would be better as a span that also logs messages.
We should then add spans around each level of a user request, for example around these lines:
- https://github.com/DiamondLightSource/FastCS/blob/e8d59fbf1b342c0d9f21d2c31675b0776276cd50/src/fastcs/transports/epics/ca/ioc.py#L222
- https://github.com/DiamondLightSource/FastCS/blob/e8d59fbf1b342c0d9f21d2c31675b0776276cd50/src/fastcs/attributes/attr_w.py#L57
- https://github.com/DiamondLightSource/FastCS/blob/e8d59fbf1b342c0d9f21d2c31675b0776276cd50/src/fastcs/attributes/attr_w.py#L65
Then drivers should add calls to their IOs as well.
Ideally we would have some unique key to correlate traces with ophyd-async, but it is not clear how that would be passed when setting PVs.
## Acceptance Criteria
- Export timings for call stack of user requests
Contributor guide
Research direction
Start by locating the Tracer implementation and its existing log_event behavior, then read wait_for_predicate in src/fastcs/attributes/attr_r.py. Trace the request path through the cited lines in src/fastcs/transports/epics/ca/ioc.py and src/fastcs/attributes/attr_w.py, including the driver IO layer. Done means user-request call stacks expose exported timings through OpenTelemetry, with the logging and correlation behavior resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100