microsoft / microsoft/agent-framework-go

Gemini: set includeServerSideToolInvocations when mixing function tools with native tools (needs backend-detection decision)

Open
#1,102 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
618
Forks
56
Avg merge
1d 1h
Merged PRs (30d)
122

Description

Porting parity item from the Python Gemini client, surfaced while comparing `provider/geminiprovider` against `python/packages/gemini`.

## Gap

On the Gemini **Developer API**, when a request combines `FunctionDeclarations` with a server-side/native tool (Google Search, code execution, file search, …), the Python client asks the server to echo its tool interactions:

```python
# _prepare_config
if tools and not self._vertexai and self._requires_server_side_tool_invocations(tools):
tool_config = tool_config or types.ToolConfig()
tool_config.include_server_side_tool_invocations = True
```

The Go provider never sets `genai.ToolConfig.IncludeServerSideToolInvocations` (the field exists — `types.go:2865` — but is unreferenced in the provider). Combined with the fact that Go also drops server-side `toolCall`/`toolResponse` response parts today (separate PR), a Go caller mixing a function tool with e.g. web search never observes the server-side tool invocations that Python surfaces.

## Why this needs a decision before a PR

Python gates the flag on `not self._vertexai`, because the field is **Developer-API-only** — the genai SDK itself errors if it is sent in Vertex mode (`models.go`). The Go provider cannot reproduce that gate cleanly:

- `genai.Client` stores its backend in the **unexported** `clientConfig` field and exposes **no accessor** (`func (c *Client) ...`) to read the backend or `ClientConfig`.
- So the provider has no supported way to know whether it is talking to the Developer API or Vertex AI at request-build time.

## Options

1. Add an `AgentConfig` field (e.g. `Backend genai.Backend`, or a narrower `UseVertexAI bool`) so callers can declare the backend the provider was constructed against, and gate the flag on it — mirroring Python’s `_vertexai`.
2. Only set the flag when the caller has explicitly opted in via a new config flag (no auto-detection), leaving backend correctness to the caller.
3. Upstream a backend accessor on `genai.Client` and gate on that (out of this repo’s control; slowest).

Option 1 seems closest to the .NET/Python behavior and to how `AgentConfig` already carries provider-construction context. Happy to implement whichever direction maintainers prefer — filing first per CONTRIBUTING since this adds public config surface.

## References
- Python: `python/packages/gemini/agent_framework_gemini/_chat_client.py` — `_prepare_config`, `_requires_server_side_tool_invocations`.
- SDK field: `google.golang.org/genai` `ToolConfig.IncludeServerSideToolInvocations` (Developer-API only).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Go provider construction and AgentConfig, then compare provider/geminiprovider with Python's _chat_client.py _prepare_config and _requires_server_side_tool_invocations. Read types.go:2865 and models.go to confirm the Developer-API restriction and backend-access limitation. Done means maintainers choose a backend-detection or opt-in approach before implementation proceeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.