process-spans: TypeError comparing evalsha_latency_ms (bytes) with float when using Dragonfly as Redis
- Dominant language
- Python
- Stars
- 44.8k
- Forks
- 4.9k
- Avg merge
- 21h 10m
- Merged PRs (30d)
- 635
Description
### Environment
Self-hosted Sentry via [sentry-kubernetes/charts](https://github.com/sentry-kubernetes/charts) **v31.5.0**, application image **26.5.0** (also reproduced on **26.2.1**).
- **Redis backend:** [Dragonfly](https://www.dragonflydb.io/) (Redis-compatible), not stock Redis
- **Connection:** `externalRedis` in Helm values — `redis://dragonfly:6379/1` with password
- **Consumer:** `sentry run consumer process-spans` (`processSpans` deployment)
Related closed issue: #80440 (Dragonfly Lua `randomseed` — different failure mode).
### Steps to Reproduce
1. Deploy Sentry with `externalRedis` pointing at Dragonfly.
2. Send performance/span data so the `ingest-spans` Kafka topic receives messages (or let backlog exist from SDK traffic).
3. Observe `process-spans` consumer pod logs.
### Expected Result
`process-spans` runs continuously, commits Kafka offsets, and performance traces appear in the UI.
### Actual Result
Consumer crashes on every batch that hits span buffer metrics handling:
```python
File "/usr/src/sentry/src/sentry/spans/buffer.py", line 473, in process_spans
if evalsha_latency_ms > longest_evalsha_data[0]:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'bytes' and 'float'
```
- Pod enters **CrashLoopBackOff**; `ingest-spans` offset does not advance.
- **Errors/issues pipeline is unaffected** (separate consumers).
- **Stock Redis:** we have not verified on Redis 7 yet; Dragonfly returns Lua numeric fields as **bytes** while Sentry expects a numeric type for `evalsha_latency_ms`.
### Product Area
Performance Monitoring / ingest (process-spans consumer, spans buffer)
### Version
26.5.0 (26.2.1 identical failure at `buffer.py` line 354)
### Suggested fix
Coerce `evalsha_latency_ms` (and similar EVALSHA return fields) to `float` when `isinstance(value, bytes)` before comparison, or document Dragonfly as unsupported for the spans buffer until Redis semantics match.
Contributor guide
Research direction
Start in src/sentry/spans/buffer.py at process_spans, especially the comparison around line 473 and the earlier line 354 path. Trace how span buffer metrics are returned from EVALSHA when using Dragonfly, then reproduce with the process-spans consumer and Dragonfly. Done means the consumer no longer raises the bytes-versus-float TypeError and can advance its Kafka offset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, python, redis
- Domain
- backend, data-engineering, databases, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100