FastAPI and httpx proxy service got dup trace_id across different requests without traceparent header.

Describe your environment

OS: (e.g, Ubuntu)
Python version: (e.g., Python 3.8.10)
Package version: (e.g., 0.46.0)

What happened?

We have write a basic fastapi proxy service which use httpx as client. We observe many duplicate trace_ids across total different requests in our product environment, but after review the code and try local, we are unable to reproduce this situation.

Currently, The bug is the second request while walk into the asgi middleware. As it call the method _start_internal_or_server_span it will get the last request's <operation_name> http send span as its parent span, so the duplicatiion happen.

We mock the middleware code to log some log like below.

Our applicaiton code is much like the below:

class Req:
    def __init__(self):
        from base.client import context_client

        self.client = httpx_client.Client(host="<some_host>")

    async def __call__(self, request: Request, path: str, timeout=360, **kwargs):
        """
        @params request: request obj
        @params path: api path

        @return : data
        """
        url = self.format_url(request, path)

        headers = request.headers.raw
        new_headers = self._fix_host(headers, url)

        requests = self.client.client.build_request(
            request.method,
            url,
            headers=new_headers,
            content=request.stream(),
            timeout=timeout,
            **kwargs,
        )
        response = await self.client.client.send(requests, stream=PROXY_STREAM)

        response.headers = transform_headers(response.headers)

        return response

    def _fix_host(self, headers: RawHeader, url: httpx.URL) -> RawHeader:
        host = f"{url.host}"
        if url.port is not None:
            host += f":{url.port}"

        for idx, (key, _) in enumerate(headers):
            if key == b"host":
                headers[idx] = (key, host.encode("utf-8"))
                break

        return headers

    def format_url(self, request: Request, path: str) -> httpx.URL:
        """
        @params request: request obj
        @params path: api path

        @return : url
        """
        full_url = self._make_url(path)
        return httpx.URL(url=full_url, query=request.url.query.encode("utf-8"))

    def _make_url(self, path: str) -> str:
        if path.startswith("/"):
            path = path[1:]

        return self.client.base_url + path


req = Req()

def relay(resp: Response):
       return StreamingResponse(
            resp.aiter_raw(),
            status_code=resp.status_code,
            headers=resp.headers,
            background=BackgroundTask(resp.aclose),
        )


@router.get("/sessions")
async def list_session_proxy(
    request: Request,
    user_id: str = Query(..., min_length=1),
):
    resp = await req(request, API_SESSIONS)
    return relay(resp)

app = FastAPI()
app.add_router(router)

We have take weeks to view the source code and add logs to try fix, but it does not work. So I just put it here for recording and ask for help.

Steps to Reproduce

Unable to reproduce local, but happens on product environments.

Expected Result

different requests should not have same trace_id.

Actual Result

the second requests get the last request asgi http send span as it internal parent span, which make the trace_id dups.

Additional context

No response

Would you like to implement a fix?

None

Start by reading the ASGI middleware path around _start_internal_or_server_span and the asgi http send span handling. Investigate how concurrent requests without a traceparent header select their parent span, then add a regression test or reproduction for the reported proxy flow. Done means separate requests receive distinct trace IDs.