DiamondLightSource / DiamondLightSource/fastcs

Add OpenTelemetry tracing of function calls from transport user input to controller IO call

Open
#155 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
8
Avg merge
2d 12h
Merged PRs (30d)
3

Description

Add `start_trace` and `stop_trace` methods to `Tracer`. Where `log_event` can be used to log trace level messages about a instantaneous point in time, these new methods can be used to trace the start and stop of some long process. These methods will start and stop spans and publish via [OpenTelemetry](https://opentelemetry.io/docs/languages/python/getting-started/) as well as publishing a log message (this could be optional so that spans can be published without logging).

For example, the two `log_event` calls in [`wait_for_predicate`](https://github.com/DiamondLightSource/FastCS/blob/c6ce4389f0cc5842d3f0e1b6327d40a69d896cdc/src/fastcs/attributes/attr_r.py#L148) would be better as a span that also logs messages.

We should then add spans around each level of a user request, for example around these lines:
- https://github.com/DiamondLightSource/FastCS/blob/e8d59fbf1b342c0d9f21d2c31675b0776276cd50/src/fastcs/transports/epics/ca/ioc.py#L222
- https://github.com/DiamondLightSource/FastCS/blob/e8d59fbf1b342c0d9f21d2c31675b0776276cd50/src/fastcs/attributes/attr_w.py#L57
- https://github.com/DiamondLightSource/FastCS/blob/e8d59fbf1b342c0d9f21d2c31675b0776276cd50/src/fastcs/attributes/attr_w.py#L65

Then drivers should add calls to their IOs as well.

Ideally we would have some unique key to correlate traces with ophyd-async, but it is not clear how that would be passed when setting PVs.

## Acceptance Criteria
- Export timings for call stack of user requests

Contributor guide

Open the contributing guide

Research direction

Start by locating the Tracer implementation and its existing log_event behavior, then read wait_for_predicate in src/fastcs/attributes/attr_r.py. Trace the request path through the cited lines in src/fastcs/transports/epics/ca/ioc.py and src/fastcs/attributes/attr_w.py, including the driver IO layer. Done means user-request call stacks expose exported timings through OpenTelemetry, with the logging and correlation behavior resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.