deepset-ai / deepset-ai/haystack
Fix async LinkContentFetcher custom client headers
@davidsbatista is already working on this.
Since Sep 19, 2026.
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 194
Description
Describe the bug
LinkContentFetcher applies headers configured through client_kwargs to
synchronous requests but not to asynchronous ones.
_get_headers() reads self._client.headers. run_async() initializes only
self._async_client, and _fetch_async() then calls _get_headers(). As a
result, default headers supplied through client_kwargs={"headers": ...} are
included by run() but are omitted by run_async().
Error message
Haystack does not raise a specific exception. A server that requires the custom
header commonly returns an HTTP error, such as 401 Unauthorized or 403 Forbidden, from run_async() while run() succeeds with the same component
configuration.
Expected behavior
run() and run_async() should send the same configured client-default
headers.
Header precedence should remain unchanged:
- Client defaults
- Haystack defaults
request_headers- Rotating
User-Agent
Additional context
Affected source: haystack/components/fetchers/link_content.py.
This affects asynchronous pipelines that pass API keys, tenant headers, or
other default headers through client_kwargs={"headers": ...}. The synchronous
path handles those headers because self._client exists; the asynchronous path
does not, because only self._async_client exists.
A likely fix is to pass the active HTTP client to the header-construction helper
or to centralize configured default headers. A regression test should assert
header parity between run() and run_async().
To Reproduce
-
Start an HTTP test server that rejects requests without
X-Test-Token. -
Configure the fetcher:
fetcher = LinkContentFetcher( client_kwargs={"headers": {"X-Test-Token": "token"}}, ) -
Call
fetcher.run(urls=[server_url])and verify that the server receives
X-Test-Token: token. -
Create a fresh fetcher with the same configuration and call:
result = await fetcher.run_async(urls=[server_url]) -
Observe that the async request omits
X-Test-Token, causing the test server
to reject the request.
FAQ Check
- Have you had a look at our new FAQ page?
System
- OS: Windows 11
- Haystack version: current checkout (commit b717d00)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.