open-telemetry / open-telemetry/opentelemetry-python-contrib

CancelledError when instrumenting FastAPI streaming endpoint running with Granian

Open
#3,327 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.1k
Forks
1.1k
Avg merge
4d 15h
Merged PRs (30d)
16

Description

Describe your environment

OS: (e.g, Ubuntu): macOs Sequoia 15.3.1
Python version: 3.11.9
Packages version:

opentelemetry-api 1.30.0
opentelemetry-exporter-otlp 1.30.0
opentelemetry-exporter-otlp-proto-common 1.30.0
opentelemetry-exporter-otlp-proto-grpc 1.30.0
opentelemetry-exporter-otlp-proto-http 1.30.0
opentelemetry-instrumentation 0.51b0
opentelemetry-instrumentation-asgi 0.51b0
opentelemetry-instrumentation-dbapi 0.51b0
opentelemetry-instrumentation-django 0.51b0
opentelemetry-instrumentation-fastapi 0.51b0
opentelemetry-instrumentation-flask 0.51b0
opentelemetry-instrumentation-psycopg2 0.51b0
opentelemetry-instrumentation-requests 0.51b0
opentelemetry-instrumentation-urllib 0.51b0
opentelemetry-instrumentation-urllib3 0.51b0
opentelemetry-instrumentation-wsgi 0.51b0
opentelemetry-proto 1.30.0
opentelemetry-resource-detector-azure 0.1.5
opentelemetry-sdk 1.30.0
opentelemetry-semantic-conventions 0.51b0
opentelemetry-util-http 0.51b0

granian 1.7.6

What happened?
[INFO] Starting granian (main PID: 93606)
[INFO] Listening at: http://127.0.0.1:8000
[INFO] Spawning worker-1 with pid: 93608
[INFO] Started worker-1
[INFO] Started worker-1 runtime-1
[ERROR] Application callable raised an exception
Traceback (most recent call last):
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/granian/_futures.py", line 11, in future_watcher
    await inner(watcher.scope, watcher.proto)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 743, in __call__
    await self.app(scope, otel_receive, otel_send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
    await response(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/responses.py", line 258, in __call__
    async with anyio.create_task_group() as task_group:
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 771, in __aexit__
    raise exc_val
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/responses.py", line 265, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/responses.py", line 261, in wrap
    await func()
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 816, in otel_receive
    message = await receive()
              ^^^^^^^^^^^^^^^
CancelledError: Future cancelled.
Steps to Reproduce
import asyncio

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

app = FastAPI()


def number_stream():
    async def generator():
        for i in range(10):
            yield f"{i}\n"
            await asyncio.sleep(1)

    return generator()


@app.get("/stream")
def stream():
    return StreamingResponse(number_stream(), media_type="text/plain")


FastAPIInstrumentor.instrument_app(app)

curl -X GET 'http://127.0.0.1:8000/stream'
Expected Result

No errors

Actual Result
[INFO] Starting granian (main PID: 93606)
[INFO] Listening at: http://127.0.0.1:8000
[INFO] Spawning worker-1 with pid: 93608
[INFO] Started worker-1
[INFO] Started worker-1 runtime-1
[ERROR] Application callable raised an exception
Traceback (most recent call last):
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/granian/_futures.py", line 11, in future_watcher
    await inner(watcher.scope, watcher.proto)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 743, in __call__
    await self.app(scope, otel_receive, otel_send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
    await response(scope, receive, send)
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/responses.py", line 258, in __call__
    async with anyio.create_task_group() as task_group:
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 771, in __aexit__
    raise exc_val
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/responses.py", line 265, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/responses.py", line 261, in wrap
    await func()
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/starlette/responses.py", line 238, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/Users/waterknight1998/Documents/granian-test/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 816, in otel_receive
    message = await receive()
              ^^^^^^^^^^^^^^^
CancelledError: Future cancelled.
Additional context

No response

Would you like to implement a fix?

None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with opentelemetry/instrumentation/asgi/init.py, especially the otel_receive path shown in the traceback, and reproduce the issue using FastAPIInstrumentor, StreamingResponse, Granian, and the supplied curl command. Determine why the receive wrapper surfaces Future cancelled during streaming; done means the reproduction completes without the reported application error.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, python
Domain
api, backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.