googleapis / googleapis/mcp-toolbox
Feature Request: Enhance Valkey Source to Support Direct Field Filtering
- Dominant language
- Go
- Stars
- 16.4k
- Forks
- 1.7k
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 85
Description
### Problem Statement
Currently, when defining a `valkey` tool within `genai-toolbox` to retrieve data based on specific field values (e.g., searching for hotels by name), the process is inefficient and burdensome for both the Valkey instance and the `genai-toolbox` client. The typical pattern involves:
1. **Fetching all relevant keys:** Using commands like `KEYS hotel:*` (or `SCAN` for production environments).
2. **Client-side iteration and data retrieval:** The `genai-toolbox` agent would then have to iterate through each key and execute `HGETALL $key` to retrieve the entire hash.
3. **Client-side filtering:** Finally, the filtering logic (e.g., matching the `name` field) is applied within the agent's code.
This approach presents several significant drawbacks:
* **Inefficient Data Transfer:** Large amounts of data are unnecessarily transferred from Valkey to the client, even if only a small subset matches the filter criteria.
* **Increased Latency:** Multiple network round trips (for `KEYS`/`SCAN` and then for each `HGETALL`) contribute to higher latency.
* **Client-Side Processing Overhead:** The filtering logic, which could ideally be handled closer to the data, consumes resources on the `genai-toolbox` client or the orchestrating agent.
* **Scalability Concerns:** For large Valkey datasets, `KEYS` can be a blocking operation, impacting server performance. Even `SCAN` followed by many `HGETALL` calls can strain the network and client.
### Proposed Solution: Direct Field Filtering Capability
We propose enhancing the `valkey` source kind within `genai-toolbox` to allow for direct, server-side field-based filtering. This would enable users to define Valkey tools that encapsulate filtering logic, making them more efficient, declarative, and easier to use within GenAI applications.
**Conceptual `tools.yaml` Example:**
```yaml
search-hotels-by-name:
kind: valkey
source: my-valkey-instance
description: |
Searches for hotels in Valkey by matching the provided keyword against hotel names.
This tool internally fetches and filters relevant hotel data on the Valkey side.
parameters:
- name: name
type: string
description: The keyword to match in hotel name (e.g., "Bern").
query:
key_pattern: "hotel:*" # Specifies the key pattern to narrow the search
filter_by_field: "name" # The field within the hash to apply the filter
filter_value: "$name" # The value to match against the specified field
# Optional: Consider supporting different match types like "exact", "contains", "regex"
# match_type: "contains"
```
### How this would work (Internal Implementation Considerations):
To implement this, the `genai-toolbox` `valkey` source could internally leverage the following approaches, prioritizing the most efficient:
1. Valkey-Search Module (Recommended):
This is the most robust and performant solution. If the Valkey instance has the `Valkey-Search` module installed, `genai-toolbox` could:
- Index Creation: Potentially provide a mechanism (e.g., a setup step or auto-detection) to ensure relevant fields (like name for hotel:) are indexed using FT.CREATE.
- Query Translation: Internally translate the filter_by_field and filter_value parameters into a Valkey-Search query (e.g., FT.SEARCH hotel_idx "@name:$name"), performing the filtering natively on the Valkey server.
- Benefits: Offers highly optimized performance for complex queries, rich query capabilities (full-text search, numeric ranges, tags, vector search), and leverages a battle-tested, dedicated search solution.
2. Server-Side Lua Scripting:
If `Valkey-Search` is not available or preferred, `genai-toolbox` could generate and execute a Lua script directly on the Valkey server. This script would encapsulate the entire filtering logic:
- It would use SCAN to efficiently iterate through keys matching the key_pattern.
- For each matched key, it would use HGETALL to retrieve the hash data.
- Crucially, it would use Lua local variables to store these intermediate results and apply the filter_by_field and filter_value logic within the script's execution environment.
- Finally, it would return only the matching results, minimizing data transfer back to the client.
- Benefits: Reduces network round trips to a single EVAL call and performs all processing server-side.
- Considerations: Writing and maintaining complex Lua scripts can be challenging, and long-running scripts can potentially block the Valkey server.
Contributor guide
Research direction
Start by locating the valkey source implementation and the tools.yaml source configuration described in the issue. Compare the proposed Valkey-Search and Lua approaches, then define which server-side filtering behavior and configuration are required; done should mean matching Valkey hash fields without client-side iteration and retrieval.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100