open-telemetry / open-telemetry/opentelemetry-python

Sporadic Connection Errors on Azure containers apps/jobs

Open
#4,632 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.6k
Forks
1k
Avg merge
4d 15h
Merged PRs (30d)
19

Description

Describe your environment

OS: Various Linux distros (different base images used)
Python version: Python 3.11/3.12
SDK version: 1.32.1

What happened?

I'm using logfire as my logging library, which is a wrapper on top of the opentelemetry sdk.
We run microservices on azure container apps/jobs.

Sometimes, especially with jobs, the logging process fails with the following error:

requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
ERROR:opentelemetry.sdk.metrics._internal.export:Exception while exporting metrics

This is very sporadic, one job can log everything successfully, one won't be able to log at all, and some would stop mid-way through the run, leading to partial spans.
I've discussed this issue with the logfire team here and they claim it's not an issue with their backend.
I'm still not sure that's 100% accurate but I thought maybe here I'll find an idea as to why it happens.

Steps to Reproduce

If it helps, here's how I setup my logfire configuration (it sets up otel behind the scenes):

logfire.configure(
	send_to_logfire="if-token-present",
	token=settings.LOGFIRE_TOKEN,
	service_name='SomeJob',
	environment=settings.ENV_TYPE,
	console=logfire.ConsoleOptions(min_log_level="trace", show_project_link=False),
	advanced=logfire.AdvancedOptions(base_url="https://logfire-api.pydantic.dev")
)
Expected Result

logging is consistent and the connection doesn't interrupt midway.

Actual Result

sporadic connection errors.

Additional context

No response

Would you like to implement a fix?

None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no repository files or tests; start by reproducing the sporadic export failure in Azure Container Apps or Jobs with the shown Python and Logfire configuration. Compare successful and failed runs, and review the linked Logfire discussion; done means identifying the cause of the connection resets and establishing consistent metric and span export.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.