elastic / elastic/apm-agent-python

opentelemetry.tracer crashes with TypeError in transaction with high number of spans

Open
#2,245 3 comments 0 reactions 0 assignees View on GitHub
agent-python community triage
Dominant language
Python
Stars
431
Forks
239
Avg merge
5d 10h
Merged PRs (30d)
7

Description

**Describe the bug**:

Using opentelemetry bridge crashes with `TypeError: 'NoneType' object does not support item assignment` exception when number of spans in transaction is higher than `ELASTIC_APM_TRANSACTION_MAX_SPANS`

```
File ~/.pyenv/versions/3.11.11/envs/test/lib/python3.11/site-packages/elasticapm/contrib/opentelemetry/trace.py:169, in Tracer.start_span(self, name, context, kind, attributes, links, start_time, record_exception, set_status_on_exception)
167 span.set_attributes(attributes)
168 spankind = get_span_kind(kind)
--> 169 elastic_span.context["otel_spankind"] = spankind
171 return span
````
.
Debugger shows that `elastic_span` inside is `DroppedSpan`. Increasing nr of `ELASTIC_APM_TRANSACTION_MAX_SPANS` "fixes" problem as shown in code below.

**To Reproduce**

```python
from elasticapm.contrib.opentelemetry import trace
from elasticapm import Client, capture_span

import os

# Reduce number of spans required to reproduce the issue
ELASTIC_APM_TRANSACTION_MAX_SPANS = 10
STEPS_ABOVE = ELASTIC_APM_TRANSACTION_MAX_SPANS + 1
STEPS_BELLOW = ELASTIC_APM_TRANSACTION_MAX_SPANS - 1
os.environ["ELASTIC_APM_TRANSACTION_MAX_SPANS"] = str(ELASTIC_APM_TRANSACTION_MAX_SPANS)

apm_client = Client()
tracer = trace.get_tracer(__name__)

# Simulate deep nested spans
def capture_span_elasticapm(steps=10):
if steps == 0:
return

with capture_span():
capture_span_elasticapm(steps-1)

def capture_span_opentelemetry(steps=10):
if steps == 0:
return

with tracer.start_as_current_span("test"):
capture_span_opentelemetry(steps-1)

# TEST

# Works - Expected
transaction = apm_client.begin_transaction('processors')
capture_span_elasticapm(STEPS_BELLOW)
apm_client.end_transaction('myapp.billing_process')

# Works - Expected
transaction = apm_client.begin_transaction('processors')
capture_span_opentelemetry(STEPS_BELLOW)
apm_client.end_transaction('myapp.billing_process')

# Works - Expected
transaction = apm_client.begin_transaction('processors')
capture_span_elasticapm(STEPS_ABOVE)
apm_client.end_transaction('myapp.billing_process')

# Crashes - Unexpected
# TypeError: 'NoneType' object does not support item assignment
transaction = apm_client.begin_transaction('processors')
capture_span_opentelemetry(STEPS_ABOVE)
apm_client.end_transaction('myapp.billing_process')
```

**Environment (please complete the following information)**
- OS: [e.g. Linux] Linux Mint 22.1 Cinnamon
- Python version: `3.11.11`
- Framework and version [e.g. Django 2.1]: N/A
- APM Server version: N/A
- Agent version: `elastic-apm==6.23.0`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.