open-telemetry / open-telemetry/opentelemetry-python

Open telemetry auto instrumentation log flush error

Open
#4,697 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.6k
Forks
1k
Avg merge
4d 15h
Merged PRs (30d)
19

Description

Describe your environment

OS: Windows 11
Python version: 3.11.9
SDK version: 1.35.0
API version: 1.35.0

What happened?

I have a very basic FastAPI that I am orchestrating with dotnet aspire (I don't think the aspire part is related, but mentioning it to be sure). I am using the zero code instrumentation to export all logs, traces, and metrics to the aspire dashboard, however, in log,s I keep getting this error non-stop:

Traceback (most recent call last):
  File "D:\repos\...\PythonApp\.venv\Lib\site-packages\opentelemetry\sdk\_shared_internal\__init__.py", line 152, in _export
    self._exporter.export(
  File "D:\repos\...\PythonApp\.venv\Lib\site-packages\opentelemetry\sdk\_logs\_internal\export\__init__.py", line 98, in export
    self.out.flush()
OSError: [Errno 22] Invalid argument

The error starts happening shortly after the app starts, and even before the app receives a single HTTP request.

I can see that the app is started with this command:

D:\repos...\PythonApp.venv\Scripts\opentelemetry-instrument.exe --traces_exporter otlp --logs_exporter console,otlp --metrics_exporter otlp
D:\repos...\PythonApp.venv\Scripts\python.exe main.py

I have some auto instrumentation libraries installed such as: "opentelemetry-instrumentation-fastapi". I can paste them all here if it will be useful. I am not using any manual instrumentation.

To be clear: everything is still working, both the console logger and the aspire dashboard. I can see all the traces, metrics, and logs in the dashboard.

And this error only appears on the Aspire dashboard, where the structured logs are supposed to appear, and not in the console, and that makes sense since the error seems to be coming from the exporter.

Steps to Reproduce

Start an auto-instrumented FastAPI that exports logs using grpc protocol to some compatible visualizer such as Grafana, Sentry, Azure App Insights or the Dotnet Aspire Dashboard (I only tested it with the Aspire dashboard, but I think it would be the same in other ones too)

Expected Result

I don't want to see the errors showing up every time it tries to flush. Everything is still working, but it's pretty annoying since it makes it impossible to see anything else on the dashboard.

Actual Result

Non-stop error message from the log exporter.

Additional context

No response

Would you like to implement a fix?

None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in opentelemetry/sdk/_logs/_internal/export/init.py at the exporter's self.out.flush() call, then reproduce the issue with an auto-instrumented FastAPI app using the grpc logs exporter and console,otlp configuration. Done means the exporter continues sending logs to the dashboard without repeatedly emitting the OSError during flush.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, grpc, python
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.