typesense / typesense/typesense

Adding nested fields with type `auto` causes document indexing errors

Open
#1,905 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage
Dominant language
C++
Stars
26.6k
Forks
973
Avg merge
18h 45m
Merged PRs (30d)
4

Description

Description

I've encountered an issue when adding nested fields of type auto. Creating all the nested type auto fields when the collection is created works, but creating one in a separate operation seems to cause field type errors. I'm not sure if I'm missing something or if this is a bug.

Steps to reproduce

  1. Create a collection with a nested field of type auto e.g:

    curl "https://****.typesense.net/collections" \
      -X POST \
      -H "Content-Type: application/json" \
      -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
      -d '{
            "enable_nested_fields": true,
            "fields": [
              {
                "name": "name",
                "type": "string"
              },
              {
                "facet": true,
                "name": "nested.foo",
                "optional": true,
                "type": "auto"
              }
            ],
            "name": "test_collection"
          }'
    

    Response:

    {
      "created_at": 1724205875,
      "default_sorting_field": "",
      "enable_nested_fields": true,
      "fields": [
        {
          "facet": false,
          "index": true,
          "infix": false,
          "locale": "",
          "name": "name",
          "optional": false,
          "sort": false,
          "stem": false,
          "store": true,
          "type": "string"
        },
        {
          "facet": true,
          "index": true,
          "infix": false,
          "locale": "",
          "name": "nested.foo",
          "optional": true,
          "sort": false,
          "stem": false,
          "store": true,
          "type": "auto"
        }
      ],
      "name": "test_collection",
      "num_documents": 0,
      "symbols_to_index": [],
      "token_separators": []
    }
    
  2. Upload a document that adheres to the schema:

    curl "https://****.typesense.net/collections/test_collection/documents" \
      -X POST \
      -H "Content-Type: application/json" \
      -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
      -d '{
            "name": "mydoc",
            "nested": {
              "foo": "hello"
            }
          }'
    

    This is successfully inserted:

    {
      "id": "0",
      "name": "mydoc",
      "nested": {
        "foo": "hello"
      }
    }
    
  3. Update the collection with another "sibling" nested field of type auto:

    curl "https://****.typesense.net/collections/test_collection" \
      -X PATCH \
      -H "Content-Type: application/json" \
      -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
      -d '{
            "fields": [
              {
                "name": "nested.bar",
                "type": "auto",
                "optional": true,
                "facet": true
              }
            ]
          }'
    

    Response:

    {
      "fields": [
        {
          "facet": true,
          "name": "nested.bar",
          "optional": true,
          "type": "auto"
        }
      ]
    }
    
  4. Upload another document that adheres to the new schema:

    curl "https://****.typesense.net/collections/test_collection/documents" \
      -X POST \
      -H "Content-Type: application/json" \
      -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
      -d '{
            "name": "indeedadoc",
            "nested": {
              "foo": "hello",
              "bar": "world"
            }
          }'
    

    The document fails to index with an error about nested.bar's type:

    {
      "message": "Field `nested.bar` has an incorrect type."
    }
    

This does work as expected if I create both nested fields at once, however:

  1. Create the collection with both nested fields in the schema:

    curl "https://****.typesense.net/collections" \
      -X POST \
      -H "Content-Type: application/json" \
      -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
      -d '{
            "enable_nested_fields": true,
            "fields": [
              {
                "name": "name",
                "type": "string"
              },
              {
                "facet": true,
                "name": "nested.foo",
                "optional": true,
                "type": "auto"
              },
              {
                "facet": true,
                "name": "nested.bar",
                "optional": true,
                "type": "auto"
              }
            ],
            "name": "test_collection2"
          }'
    

    Result:

    {
      "created_at": 1724205997,
      "default_sorting_field": "",
      "enable_nested_fields": true,
      "fields": [
        {
          "facet": false,
          "index": true,
          "infix": false,
          "locale": "",
          "name": "name",
          "optional": false,
          "sort": false,
          "stem": false,
          "store": true,
          "type": "string"
        },
        {
          "facet": true,
          "index": true,
          "infix": false,
          "locale": "",
          "name": "nested.foo",
          "optional": true,
          "sort": false,
          "stem": false,
          "store": true,
          "type": "auto"
        },
        {
          "facet": true,
          "index": true,
          "infix": false,
          "locale": "",
          "name": "nested.bar",
          "optional": true,
          "sort": false,
          "stem": false,
          "store": true,
          "type": "auto"
        }
      ],
      "name": "test_collection2",
      "num_documents": 0,
      "symbols_to_index": [],
      "token_separators": []
    }
    
  2. Insert a document that conforms to the schema:

    curl "https://****.typesense.net/collections/test_collection2/documents" \
      -X POST \
      -H "Content-Type: application/json" \
      -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
      -d '{
            "name": "indeedadoc",
            "nested": {
              "foo": "hello",
              "bar": "world"
            }
          }'
    

    The document inserts successfully:

    {
      "id": "0",
      "name": "indeedadoc",
      "nested": { "bar": "world", "foo": "hello" }
    }
    

The two collection schemas are identical when fetched, other than name and created_at:

curl "https://****.typesense.net/collections/test_collection" \
  -X GET \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}"

Result:

{
  "created_at": 1724205875,
  "default_sorting_field": "",
  "enable_nested_fields": true,
  "fields": [
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "name",
      "optional": false,
      "sort": false,
      "stem": false,
      "store": true,
      "type": "string"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "nested.foo",
      "optional": true,
      "sort": false,
      "stem": false,
      "store": true,
      "type": "auto"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "nested.bar",
      "optional": true,
      "sort": false,
      "stem": false,
      "store": true,
      "type": "auto"
    }
  ],
  "name": "test_collection",
  "num_documents": 1,
  "symbols_to_index": [],
  "token_separators": []
}
curl "https://****.typesense.net/collections/test_collection2" \
  -X GET \
  -H "Content-Type: application/json" \
  -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}"

Result:

{
  "created_at": 1724205997,
  "default_sorting_field": "",
  "enable_nested_fields": true,
  "fields": [
    {
      "facet": false,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "name",
      "optional": false,
      "sort": false,
      "stem": false,
      "store": true,
      "type": "string"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "nested.foo",
      "optional": true,
      "sort": false,
      "stem": false,
      "store": true,
      "type": "auto"
    },
    {
      "facet": true,
      "index": true,
      "infix": false,
      "locale": "",
      "name": "nested.bar",
      "optional": true,
      "sort": false,
      "stem": false,
      "store": true,
      "type": "auto"
    }
  ],
  "name": "test_collection2",
  "num_documents": 1,
  "symbols_to_index": [],
  "token_separators": []
}

Expected Behavior

Indexing a document should work the same way when a nested field with type auto is either:

  • Present when the collection is initially created
  • Added after the collection is created

Actual Behavior

Adding a document fails when an nested auto field is added after a collection is created.

Metadata

Typesense Version: v27.0.rc34

OS: Typesense Cloud

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the curl reproduction in the issue, comparing a nested auto field declared at collection creation with one added through the collection PATCH endpoint. Trace the nested-field schema and document-indexing paths for the mismatch. Done means documents containing the newly added nested auto field index successfully, matching the behavior of fields declared initially.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.