googleapis / googleapis/mcp-toolbox

Feature Request: Enhance Valkey Source to Support Direct Field Filtering

Open
#905 3 comments 0 reactions 0 assignees View on GitHub
priority: p3 type: feature request type: question
Dominant language
Go
Stars
16.4k
Forks
1.7k
Avg merge
4d 9h
Merged PRs (30d)
85

Description

### Problem Statement

Currently, when defining a `valkey` tool within `genai-toolbox` to retrieve data based on specific field values (e.g., searching for hotels by name), the process is inefficient and burdensome for both the Valkey instance and the `genai-toolbox` client. The typical pattern involves:

1. **Fetching all relevant keys:** Using commands like `KEYS hotel:*` (or `SCAN` for production environments).
2. **Client-side iteration and data retrieval:** The `genai-toolbox` agent would then have to iterate through each key and execute `HGETALL $key` to retrieve the entire hash.
3. **Client-side filtering:** Finally, the filtering logic (e.g., matching the `name` field) is applied within the agent's code.

This approach presents several significant drawbacks:

* **Inefficient Data Transfer:** Large amounts of data are unnecessarily transferred from Valkey to the client, even if only a small subset matches the filter criteria.
* **Increased Latency:** Multiple network round trips (for `KEYS`/`SCAN` and then for each `HGETALL`) contribute to higher latency.
* **Client-Side Processing Overhead:** The filtering logic, which could ideally be handled closer to the data, consumes resources on the `genai-toolbox` client or the orchestrating agent.
* **Scalability Concerns:** For large Valkey datasets, `KEYS` can be a blocking operation, impacting server performance. Even `SCAN` followed by many `HGETALL` calls can strain the network and client.

### Proposed Solution: Direct Field Filtering Capability

We propose enhancing the `valkey` source kind within `genai-toolbox` to allow for direct, server-side field-based filtering. This would enable users to define Valkey tools that encapsulate filtering logic, making them more efficient, declarative, and easier to use within GenAI applications.

**Conceptual `tools.yaml` Example:**

```yaml
search-hotels-by-name:
kind: valkey
source: my-valkey-instance
description: |
Searches for hotels in Valkey by matching the provided keyword against hotel names.
This tool internally fetches and filters relevant hotel data on the Valkey side.
parameters:
- name: name
type: string
description: The keyword to match in hotel name (e.g., "Bern").
query:
key_pattern: "hotel:*" # Specifies the key pattern to narrow the search
filter_by_field: "name" # The field within the hash to apply the filter
filter_value: "$name" # The value to match against the specified field
# Optional: Consider supporting different match types like "exact", "contains", "regex"
# match_type: "contains"
```
### How this would work (Internal Implementation Considerations):

To implement this, the `genai-toolbox` `valkey` source could internally leverage the following approaches, prioritizing the most efficient:

1. Valkey-Search Module (Recommended):
This is the most robust and performant solution. If the Valkey instance has the `Valkey-Search` module installed, `genai-toolbox` could:

- Index Creation: Potentially provide a mechanism (e.g., a setup step or auto-detection) to ensure relevant fields (like name for hotel:) are indexed using FT.CREATE.

- Query Translation: Internally translate the filter_by_field and filter_value parameters into a Valkey-Search query (e.g., FT.SEARCH hotel_idx "@name:$name"), performing the filtering natively on the Valkey server.

- Benefits: Offers highly optimized performance for complex queries, rich query capabilities (full-text search, numeric ranges, tags, vector search), and leverages a battle-tested, dedicated search solution.

2. Server-Side Lua Scripting:
If `Valkey-Search` is not available or preferred, `genai-toolbox` could generate and execute a Lua script directly on the Valkey server. This script would encapsulate the entire filtering logic:

- It would use SCAN to efficiently iterate through keys matching the key_pattern.

- For each matched key, it would use HGETALL to retrieve the hash data.

- Crucially, it would use Lua local variables to store these intermediate results and apply the filter_by_field and filter_value logic within the script's execution environment.

- Finally, it would return only the matching results, minimizing data transfer back to the client.

- Benefits: Reduces network round trips to a single EVAL call and performs all processing server-side.

- Considerations: Writing and maintaining complex Lua scripts can be challenging, and long-running scripts can potentially block the Valkey server.

Contributor guide

Open the contributing guide

Research direction

Start by locating the valkey source implementation and the tools.yaml source configuration described in the issue. Compare the proposed Valkey-Search and Lua approaches, then define which server-side filtering behavior and configuration are required; done should mean matching Valkey hash fields without client-side iteration and retrieval.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.