getsentry / getsentry/sentry

process-spans: TypeError comparing evalsha_latency_ms (bytes) with float when using Dragonfly as Redis

Open Beginner friendly
#116,116 5 comments 0 reactions 0 assignees View on GitHub
Feature Product Area: Performance
Dominant language
Python
Stars
44.8k
Forks
4.9k
Avg merge
21h 10m
Merged PRs (30d)
635

Description

### Environment

Self-hosted Sentry via [sentry-kubernetes/charts](https://github.com/sentry-kubernetes/charts) **v31.5.0**, application image **26.5.0** (also reproduced on **26.2.1**).

- **Redis backend:** [Dragonfly](https://www.dragonflydb.io/) (Redis-compatible), not stock Redis
- **Connection:** `externalRedis` in Helm values — `redis://dragonfly:6379/1` with password
- **Consumer:** `sentry run consumer process-spans` (`processSpans` deployment)

Related closed issue: #80440 (Dragonfly Lua `randomseed` — different failure mode).

### Steps to Reproduce

1. Deploy Sentry with `externalRedis` pointing at Dragonfly.
2. Send performance/span data so the `ingest-spans` Kafka topic receives messages (or let backlog exist from SDK traffic).
3. Observe `process-spans` consumer pod logs.

### Expected Result

`process-spans` runs continuously, commits Kafka offsets, and performance traces appear in the UI.

### Actual Result

Consumer crashes on every batch that hits span buffer metrics handling:

```python
File "/usr/src/sentry/src/sentry/spans/buffer.py", line 473, in process_spans
if evalsha_latency_ms > longest_evalsha_data[0]:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'bytes' and 'float'
```

- Pod enters **CrashLoopBackOff**; `ingest-spans` offset does not advance.
- **Errors/issues pipeline is unaffected** (separate consumers).
- **Stock Redis:** we have not verified on Redis 7 yet; Dragonfly returns Lua numeric fields as **bytes** while Sentry expects a numeric type for `evalsha_latency_ms`.

### Product Area

Performance Monitoring / ingest (process-spans consumer, spans buffer)

### Version

26.5.0 (26.2.1 identical failure at `buffer.py` line 354)

### Suggested fix

Coerce `evalsha_latency_ms` (and similar EVALSHA return fields) to `float` when `isinstance(value, bytes)` before comparison, or document Dragonfly as unsupported for the spans buffer until Redis semantics match.

Contributor guide

Open the contributing guide

Research direction

Start in src/sentry/spans/buffer.py at process_spans, especially the comparison around line 473 and the earlier line 354 path. Trace how span buffer metrics are returned from EVALSHA when using Dragonfly, then reproduce with the process-spans consumer and Dragonfly. Done means the consumer no longer raises the bytes-versus-float TypeError and can advance its Kafka offset.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, python, redis
Domain
backend, data-engineering, databases, performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.