Token incorrectly dropped when stemmed field has higher weight than non-stemmed matching field

Bug Description

When searching across multiple fields where:

• A stemmed field (stem: true) has a higher weight
• A non-stemmed field (stem: false) contains the actual match with a lower weight

Typesense incorrectly drops tokens from the query, resulting in num_tokens_dropped: 1 when it should be 0, sometimes. Note: sometimes because this happens with "james fishback" but not "john smith"

The bug does not occur when:

• Both fields have stem: true
• Both fields have stem: false
• The matching field has equal or higher weight

Reproduction Steps

#!/bin/bash
set -x

# 1. Define Variables
export TYPESENSE_API_KEY=xyz
export TYPESENSE_HOST=http://localhost:8108

# 2. Cleanup previous runs
docker stop typesense-repro 2>/dev/null
docker rm typesense-repro 2>/dev/null
rm -rf "$(pwd)"/typesense-data-dir-repro
mkdir "$(pwd)"/typesense-data-dir-repro

# 3. Start Typesense v30.0.rca34
docker run -d -p 8108:8108 --name typesense-repro \
            -v"$(pwd)"/typesense-data-dir-repro:/data \
            typesense/typesense:30.0.rca34 \
            --data-dir /data \
            --api-key=$TYPESENSE_API_KEY \
            --enable-cors

# 4. Wait for Typesense to be healthy
echo "Waiting for Typesense..."
until curl -s -o /dev/null -w "%{http_code}" "$TYPESENSE_HOST/health" -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | grep -q "200"; do
  sleep 2
done
echo "Typesense is ready!"

# 5. Create Collection - note title has stem:true, names does not
curl -s "$TYPESENSE_HOST/collections" \
       -X POST \
       -H "Content-Type: application/json" \
       -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
       -d '{
             "name": "test_collection",
             "fields": [
               {"name": "id", "type": "string"},
               {"name": "title", "type": "string", "stem": true},
               {"name": "names", "type": "string[]"}
             ]
           }' | jq

# 6. Import Document - "James Fishback" is in names, NOT in title
curl -s "$TYPESENSE_HOST/collections/test_collection/documents/import?action=create" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -H "Content-Type: text/plain" \
        -X POST \
        -d '{"id": "1", "title": "Best candidate for the position", "names": ["Jane Doe", "James Fishback", "Bob Wilson"]}' | jq

echo ""
echo "================================================================"
echo "BUG DEMONSTRATION: Token dropped when stemmed field has higher weight"
echo "================================================================"

echo ""
echo "=== TEST 1: names only - WORKS ==="
echo "Expected: num_tokens_dropped: 0, tokens_matched: 2"
curl -s "$TYPESENSE_HOST/multi_search" \
        -X POST \
        -H "Content-Type: application/json" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -d '{
              "searches": [{
                  "collection": "test_collection",
                  "q": "james fishback",
                  "query_by": "names",
                  "num_typos": "0"
              }]
            }' | jq '.results[0].hits[0].text_match_info'

echo ""
echo "=== TEST 2: names (weight 1) + title/stem (weight 2) - BUG ==="
echo "Expected: num_tokens_dropped: 0, tokens_matched: 2"
echo "Actual: num_tokens_dropped: 1, tokens_matched: 1 (BUG!)"
curl -s "$TYPESENSE_HOST/multi_search" \
        -X POST \
        -H "Content-Type: application/json" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -d '{
              "searches": [{
                  "collection": "test_collection",
                  "q": "james fishback",
                  "query_by": "names,title",
                  "query_by_weights": "1,2",
                  "num_typos": "0"
              }]
            }' | jq '.results[0].hits[0].text_match_info'

echo ""
echo "=== TEST 3: names (weight 2) + title/stem (weight 1) - WORKS ==="
echo "Expected: num_tokens_dropped: 0, tokens_matched: 2"
curl -s "$TYPESENSE_HOST/multi_search" \
        -X POST \
        -H "Content-Type: application/json" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -d '{
              "searches": [{
                  "collection": "test_collection",
                  "q": "james fishback",
                  "query_by": "names,title",
                  "query_by_weights": "2,1",
                  "num_typos": "0"
              }]
            }' | jq '.results[0].hits[0].text_match_info'

echo ""
echo "=== TEST 4: Equal weights (15,15) - WORKS ==="
echo "Expected: num_tokens_dropped: 0, tokens_matched: 2"
curl -s "$TYPESENSE_HOST/multi_search" \
        -X POST \
        -H "Content-Type: application/json" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -d '{
              "searches": [{
                  "collection": "test_collection",
                  "q": "james fishback",
                  "query_by": "names,title",
                  "query_by_weights": "15,15",
                  "num_typos": "0"
              }]
            }' | jq '.results[0].hits[0].text_match_info'

echo ""
echo "=== TEST 5: Same weights but with stem:false - WORKS ==="
echo "Recreating collection with title stem:false..."
curl -s "$TYPESENSE_HOST/collections/test_collection" -X DELETE -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" > /dev/null
curl -s "$TYPESENSE_HOST/collections" \
       -X POST \
       -H "Content-Type: application/json" \
       -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
       -d '{
             "name": "test_collection",
             "fields": [
               {"name": "id", "type": "string"},
               {"name": "title", "type": "string", "stem": false},
               {"name": "names", "type": "string[]"}
             ]
           }' > /dev/null
curl -s "$TYPESENSE_HOST/collections/test_collection/documents/import?action=create" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -H "Content-Type: text/plain" \
        -X POST \
        -d '{"id": "1", "title": "Best candidate for the position", "names": ["Jane Doe", "James Fishback", "Bob Wilson"]}' > /dev/null
echo "Expected: num_tokens_dropped: 0, tokens_matched: 2"
curl -s "$TYPESENSE_HOST/multi_search" \
        -X POST \
        -H "Content-Type: application/json" \
        -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
        -d '{
              "searches": [{
                  "collection": "test_collection",
                  "q": "james fishback",
                  "query_by": "names,title",
                  "query_by_weights": "1,2",
                  "num_typos": "0"
              }]
            }' | jq '.results[0].hits[0].text_match_info'

echo ""
echo "================================================================"
echo "SUMMARY:"
echo "- Bug occurs when stemmed field (stem:true) has higher weight"
echo "  than the field containing the actual match"
echo "- Bug does NOT occur when higher-weighted field has stem:false"
echo "- Workarounds:"
echo "  1. Set stem:false on higher-weighted fields"
echo "  2. Give matching field equal or higher weight"
echo "================================================================"

Expected vs Actual

Expected behavior
For test 2, should not drop token since its an exact match on "james fishback"

Actual behavior
Test 2 dropped 1 token, only matched on "fishback"

Environment

• Typesense version: e.g. typesense:30.0.rca34
• Operating system: e.g. Ubuntu 22.04 x86_64, macOS Sequioia 15.7.3

Start by running the reproduction script with Typesense 30.0.rca34 and compare the multi-search results for the weighted stemmed and non-stemmed fields. Trace the search path responsible for token dropping and verify the fix against the listed cases, especially the expected text_match_info with num_tokens_dropped: 0 and tokens_matched: 2.