[Demo]Function Calling Exploration with Gemma 3 — Multimodal ReAct Agent, OpenAPI Support,OS Agentic/Tool calling Frameworks

**Title:** 🔧 Function Calling Exploration with Gemma 3 — Multimodal ReAct Agent, OpenAPI Support, Web UI & Benchmarking

I have developed a complete **Python SDK for Function Calling with Gemma 3**, designed to transform Gemma into a ReAct-style **agentic system** that can interact with diverse APIs (text & image), reason across multiple turns, and deliver actionable results. The SDK draws inspiration from Berkeley’s [Gorilla](https://arxiv.org/abs/2305.15334) and Google's Gemini API, and is designed with extensibility, clarity, and real-world use in mind.

** Currently I have prepared a repository that is a skeletal code that begins to explore how function calling can be engineered as a proper solution and explores various differnent types of Agent Calling Frameworks in addition to the wrapper that i have built.

** Note - Currently i have a experiment with Gemma 4B model due to Hardware constraits however , the architecture is expandable to other variants under Gemma 3 Umbrealla.

📂 **Repo**: [Link](https://github.com/prasad-yashdeep/gemma-function-calling)

---

### 🚀 What I Built

**Core Features:**
- 🔌 **Multi-API Function Calling:** Enable Gemma to invoke multiple APIs or tools per query.
- 🧠 **Multi-Turn ReAct Reasoning:** Implements a loop of `Thought → Action → Observation` to allow step-by-step function execution.
- 📷 **Multimodal Tool Support:** Use image generation, captioning, classification APIs alongside text-based tools.
- 🧾 **OpenAPI Onboarding:** Developers can onboard REST APIs using OpenAPI specs – auto-converting them into callable tools for the model.
- 🧪 **Benchmarking Suite:** Inspired by the **Berkeley Function-Calling Leaderboard (BFCL)** to evaluate accuracy, latency, token usage.
- 🖥️ **Optional Web UI (Gorilla-style):** Built-in Gradio UI for testing queries, inspecting model reasoning, and previewing images.
- 🔄 **Model Backend Abstraction:** Switch easily between Gemma 4B local/HF-hosted, and other LLMs via config.

---

### Architecture

The SDK follows a modular architecture with three main components:

```
Existing API Schema (OpenAPI, REST Docs)
│
▼
api_converter.py (Gemma JSON definition generator)
│
▼
SDK Function Definition (Python module)
│
▼
Gemma SDK runtime (multi-turn, ReAct execution)
```

### Function Calling Implementation

Unlike some other models with native function calling tokens, our SDK implements function calling with Gemma through carefully engineered prompts and response parsing. I have developed two main approaches:

1. **JSON-based Function Calling**: Structured format for simple API interactions
2. **ReAct-style Function Calling**: Reasoning and Acting pattern for complex, multi-step tasks

### 🎯 Why It Matters

Gemma 4B is powerful and lightweight, but lacks native function calling. This SDK bridges that gap by providing:
- ✨ Prompt-driven function calling using few-shot ReAct examples.
- 📚 Dynamic tool injection based on OpenAPI specs.
- 🔄 Stateful, multi-turn orchestration that mimics agents.
- 🖼️ First-class image support via external tool abstraction.
- ⚙️ Extensible, production-ready code modules for research and deployment.

---

### 🧩 Technical Breakdown

- **FunctionRegistry** manages all callable tools (Python/local or REST).
- **OpenAPI Loader** parses OpenAPI/Swagger specs into callable functions.
- **Conversation Orchestrator** manages ReAct-style reasoning loop.
- **ModelClient** abstracts local vs. API-backed model calls (supports Gemma, GPT, etc).
- **Web UI** for test-driving function calls, image tools, and debugging.
- **Benchmarking Framework** for evaluating function use accuracy, parameter correctness, latency, and robustness.

---

### 📸 Example Use Case

**Prompt:**
> Who won the 2025 Champions Trophy? Show me the flag of the winning team.

**Model’s reasoning (ReAct):**
```txt
Thought: I need to first find the winner of the 2025 Champions Trophy.
Action: use cricket_api.get_final_match_info("ICC Champions Trophy", 2025)
Observation: {"winner": "India", ...}

Thought: Now I will fetch the flag of India.
Action: use generate_image(prompt="Indian flag")
Observation: Image generated successfully.

Final Answer: India won the 2025 Champions Trophy. Here is the flag.
```

---

### 📊 Benchmarks

We are integrating the **BFCL dataset** to benchmark:
- Function selection accuracy
- Parameter extraction correctness
- Latency (model + function time)
- Token usage
- Robustness to irrelevant prompts

This enables reproducible evaluation and comparisons across models (e.g., Gemma 4B vs Gemma 27B vs GPT-4).

---

### 📌 Roadmap

✅ Function-calling SDK with ReAct loop
✅ OpenAPI auto-onboarding
✅ Gradio-based test UI
✅ Image generation/classification tool integration
✅ Benchmarking suite w/ BFCL test loader
✅ DAG-based API Orchestration: Enable complex workflows with directed acyclic graphs

🔄 Planned:
- DAG-based multi-function composition
- Function calling exploration for various different Agentic Frameworks.
- Tool retriever for prompt compression (à la Gorilla)
- Native Gemma multimodal inference for direct image input
- Tutorials and deployment recipes

---

# Exploration of Function Calling of Gemma With Differnt OS frameworks

Each framework takes a slightly different architectural approach to LLM function calling:

- **CrewAI** introduces a dedicated function-calling LLM and parses textual actions.
- **AutoGen** uses a dual-agent approach, explicitly separating decision-making from execution.
- **LangChain** provides a single-agent loop, offering either API-level or prompt-level function integration.

All three frameworks can theoretically incorporate Gemma as the agent’s "brain," thanks to support for custom or open-source models, which can be integrated via:

- **LiteLLM** connectors in CrewAI
- **Custom ModelClient** implementations in AutoGen
- **HuggingFace Transformers** integration in LangChain

When implementing Gemma in these frameworks, developers can leverage several open-source tools:

- [**LiteLLM**](https://github.com/BerriAI/litellm) (used by CrewAI) to connect Hugging Face models or local servers.
- [**Transformers Library**](https://github.com/huggingface/transformers) to load Gemma and potentially generate function-call style outputs through careful prompting.
- [**Pydantic schemas**](https://github.com/pydantic/pydantic) for robust argument parsing, a pattern employed in CrewAI and LangChain.
- Fine-tuned tool-specialized models, like [**watt-tool-8B**](https://community.crewai.com/) as ancillary agents or references to guide Gemma’s responses.

## Challenges and Solutions

Specific challenges when integrating Gemma include:

- Ensuring Gemma consistently follows structured function-call formats (natively handled by GPT/Claude).
- Managing the resource-intensive nature of locally hosted LLM inference.

Recommended strategies and workarounds to mitigate these challenges:

- **Prompt Engineering**: Carefully crafting instructions and examples to prompt Gemma into consistently producing structured tool invocation outputs.
- **Leveraging Specialized Models**: Using smaller, function-specialized models like watt-tool-8B to complement or guide Gemma's decisions on tool usage.
- **Fine-tuning**: Slightly fine-tuning Gemma on curated function-calling data to improve reliability and formatting precision.

---

## References and Resources

- [Gemma Model Announcement (HuggingFace)](https://huggingface.co/google/gemma-7b)
- [CrewAI Documentation and Community Discussions](https://community.crewai.com/)
- [Microsoft AutoGen Tool Use Tutorial](https://microsoft.github.io/autogen/)
- [LangChain Documentation on Function Calling](https://python.langchain.com/)

---