deepset-ai / deepset-ai/haystack
Cache tool results inside Agent loops (avoid duplicate identical tool calls)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 194
Description
Is your feature request related to a problem? Please describe.
Every Agent loop I run for more than ~5 steps re-calls the same tool with the same args. The model fetches a URL, reasons over the result, then fetches the same URL three steps later because the prior answer slipped out of attention. Same shape for retrievers, calculators, web search. ToolInvoker re-runs every call without checking: haystack/components/tools/tool_invoker.py has no cache layer, and haystack/tools/tool.py has no cacheable field. The cost is paid twice each time, the tool's API call plus the LLM tokens spent deciding to make it. On long agent loops this adds up to a real chunk of the run's spend, and I'd rather not pay for the second copy of work the agent already did.
Describe the solution you'd like
A small, opt-in keyed cache around ToolInvoker. Key is (tool_name, sha256(canonicalized_args_json)). Configurable TTL and scope. Per-Tool opt-in so write-effecting tools never serve stale state.
from haystack.tools import Tool, ToolCache, InMemoryToolCache
cache = ToolCache(
backend=InMemoryToolCache(),
ttl_seconds=3600,
scope="agent_run", # or "session", "global"
)
fetch_url = Tool(name="fetch_url", function=requests.get, cacheable=True)
post_slack = Tool(name="post_slack", function=slack_post, cacheable=False)
agent = Agent(chat_generator=g, tools=[fetch_url, post_slack], tool_cache=cache)
Defaults are chosen to defuse the "stale results" concern up front:
cacheable=FalseonToolby default. Caching is opt-in per tool, sopost_slackand friends can't serve a cached"ok, sent"for a message that was never actually sent.scope="agent_run"by default. No cross-user or cross-tenant leakage unless the user explicitly widens it.- Short default TTL (5 minutes). Long enough to deduplicate within an agent loop, short enough that staleness doesn't compound.
- A
ToolCacheStats(hits, misses, calls saved) is appended to the agent's run output, so users can verify the cache is doing something before flipping more tools tocacheable=True.
When no tool_cache is passed, behavior is identical to today.
Describe alternatives you've considered
- Wrap each tool function in
functools.lru_cache. Doesn't compose withTool.to_dict()serialization, can't be scoped per-Agent, gives no stats in the run output, and every tool author has to remember. - LangChain's
set_llm_cacheor LiteLLM router caching. Different layer. Those cache the LLM call; the duplicatefetch_urlstill pays the tool's API latency. - Make every tool internally idempotent. Doesn't help. The cost being saved isn't the tool's work, it's the agent re-spending tokens to re-decide to call it.
- A semantic cache over agent steps (cache by input similarity). Bigger scope, contested design, and the wrong layer for the deterministic-args case that bites in practice.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading haystack/components/tools/tool_invoker.py and haystack/tools/tool.py to understand tool execution and serialization. Then trace Agent run output handling to determine where cache scope, TTL, opt-in behavior, and ToolCacheStats fit. Done means identical cacheable calls can be deduplicated without changing behavior when no cache is configured, while non-cacheable tools remain uncached.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100