deepset-ai / deepset-ai/haystack

Fix async LinkContentFetcher custom client headers

Open
#12,817 0 comments 0 reactions 1 assignee View on GitHub

@davidsbatista is already working on this.

Since Sep 19, 2026.

Dominant language
Python
Stars
26.6k
Forks
3.2k
Avg merge
1d 3h
Merged PRs (30d)
194

Description

Describe the bug

LinkContentFetcher applies headers configured through client_kwargs to
synchronous requests but not to asynchronous ones.

_get_headers() reads self._client.headers. run_async() initializes only
self._async_client, and _fetch_async() then calls _get_headers(). As a
result, default headers supplied through client_kwargs={"headers": ...} are
included by run() but are omitted by run_async().

Error message

Haystack does not raise a specific exception. A server that requires the custom
header commonly returns an HTTP error, such as 401 Unauthorized or 403 Forbidden, from run_async() while run() succeeds with the same component
configuration.

Expected behavior

run() and run_async() should send the same configured client-default
headers.

Header precedence should remain unchanged:

  1. Client defaults
  2. Haystack defaults
  3. request_headers
  4. Rotating User-Agent

Additional context

Affected source: haystack/components/fetchers/link_content.py.

This affects asynchronous pipelines that pass API keys, tenant headers, or
other default headers through client_kwargs={"headers": ...}. The synchronous
path handles those headers because self._client exists; the asynchronous path
does not, because only self._async_client exists.

A likely fix is to pass the active HTTP client to the header-construction helper
or to centralize configured default headers. A regression test should assert
header parity between run() and run_async().

To Reproduce

  1. Start an HTTP test server that rejects requests without X-Test-Token.

  2. Configure the fetcher:

    fetcher = LinkContentFetcher(
        client_kwargs={"headers": {"X-Test-Token": "token"}},
    )
    
  3. Call fetcher.run(urls=[server_url]) and verify that the server receives
    X-Test-Token: token.

  4. Create a fresh fetcher with the same configuration and call:

    result = await fetcher.run_async(urls=[server_url])
    
  5. Observe that the async request omits X-Test-Token, causing the test server
    to reject the request.

FAQ Check

System

  • OS: Windows 11
  • Haystack version: current checkout (commit b717d00)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.