quickwit-oss / quickwit-oss/quickwit

Adding new fields to doc_mapping unexpectedly alters tag_fields behavior

Open
#5,866 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug
When adding new fields to the doc_mapping configuration, it causes unexpected changes in the values retrieved from fields defined in tag_fields.

For example:

In the current index configuration (version 0.9, index_id: spot-trade-etl_exlogs), we have tag_fields: [app_name]properly defined. However, after adding the 'uri' field to field_mappings, the way tag content is processed changes unexpectedly.

Current Behavior
Modifying doc_mapping (such as adding the uri field) inadvertently affects the retrieval mechanism of fields marked in tag_fields (app_name in this case), producing unexpected values.

Expected behavior

  1. tag_fields values should remain stable
  2. Schema modifications (add/remove fields) should not impact existing tag_fields

Configuration:

version: 0.9
index_id: spot-trade-etl_exlogs
doc_mapping:
  mode: dynamic
  dynamic_mapping:
  field_mappings:
    - name: "@timestamp"
      type: datetime
      input_formats:
        - iso8601
        - "%Y-%m-%d %H:%M:%S.%f"
      output_format: "%Y-%m-%d %H:%M:%S.%f"
      fast: true
      fast_precision: microseconds
    - name: time
      type: datetime
      input_formats:
        - "%Y-%m-%d %H:%M:%S.%f"
        - iso8601
      output_format: "%Y-%m-%d %H:%M:%S.%f"
      fast: true
      fast_precision: milliseconds
    - name: level
      type: text
      fast: true
    - name: msg
      type: text
      tokenizer: chinese_compatible
      record: position
      fieldnorms: true
      fast:
        normalizer: lowercase
    - name: log
      type: text
      tokenizer: chinese_compatible
      record: position
      fieldnorms: true
      fast:
        normalizer: lowercase
    - name: tid
      type: text
      tokenizer: raw
      fast: true
      stored: true
      record: basic
    - name: biz_monitor_ctx
      type: text
      record: position
    - name: uri
      type: text
      record: position
    - name: event
      type: text
      fast: true
      record: position
    - name: app_name
      type: text
      tokenizer: raw
      fast: true
    - name: namespace
      type: text
      fast: true
      tokenizer: raw
    - name: pod_name
      type: text
      fast: true
      record: position
    - name: node_name
      type: text
      fast: true
      tokenizer: raw
  timestamp_field: "@timestamp"
  tag_fields: [app_name]
search_settings:
  default_search_fields: [app_name, pod_name]
indexing_settings:
  commit_timeout_secs: 30
  merge_policy:
    type: "stable_log"
    min_level_num_docs: 100000
    merge_factor: 10
    max_merge_factor: 12
retention:
  period: 15 day
  schedule: hourly
{
    "split_state": "MarkedForDeletion",
    "update_timestamp": 1755676264,
    "publish_timestamp": 1755676264,
    "version": "0.9",
    "split_id": "01K3379N8EMHT3MYJ4WD9VVFQT",
    "index_uid": "spot-trade-etl_exlogs:01JZS10GJQXVHDP72FC2XJN54T",
    "partition_id": 0,
    "source_id": "_ingest-source",
    "node_id": "quickwit-indexer-1",
    "num_docs": 37103,
    "uncompressed_docs_size_in_bytes": 111738688,
    "time_range": {
      "start": 1755676201,
      "end": 1755676231
    },
    "create_timestamp": 1755676232,
    "maturity": {
      "type": "immature",
      "maturation_period_millis": 172800000
    },
    "tags": [
      "app_name!",
      "app_name:spot-trade-etl"
    ],
    "footer_offsets": {
      "start": 17010937,
      "end": 17023033
    },
    "delete_opstamp": 0,
    "num_merge_ops": 0,
    "doc_mapping_uid": "01JZS10GJPZMKRNWPP4YZBPRY5"
  },
  {
    "split_state": "Published",
    "update_timestamp": 1755676840,
    "publish_timestamp": 1755676840,
    "version": "0.9",
    "split_id": "01K337WZ9CF81Z48RG0FPXTY5B",
    "index_uid": "spot-trade-etl_exlogs:01JZS10GJQXVHDP72FC2XJN54T",
    "partition_id": 0,
    "source_id": "_ingest-source",
    "node_id": "quickwit-indexer-7",
    "num_docs": 377859,
    "uncompressed_docs_size_in_bytes": 1127572890,
    "time_range": {
      "start": 1755676750,
      "end": 1755676834
    },
    "create_timestamp": 1755676839,
    "maturity": {
      "type": "immature",
      "maturation_period_millis": 172800000
    },
    "tags": [
      "app_name!",
      "app_name:isolatedbalanceservice",
      "app_name:kafkaservice",
      "app_name:orderkafkacounterconsumer",
      "app_name:publishadaptor",
      "app_name:spotbalanceservice",
      "app_name:tradedetailservice"
    ],
    "footer_offsets": {
      "start": 169363916,
      "end": 169403055
    },
    "delete_opstamp": 0,
    "num_merge_ops": 1,
    "doc_mapping_uid": "01K337V0YBCQX3B0BDHGZBBBFM"
  },
  {
    "split_state": "MarkedForDeletion",
    "update_timestamp": 1755676236,
    "publish_timestamp": 1755676236,
    "version": "0.9",
    "split_id": "01K3379M7NH0WBHF2Z827B7FZY",
    "index_uid": "spot-trade-etl_exlogs:01JZS10GJQXVHDP72FC2XJN54T",
    "partition_id": 0,
    "source_id": "_ingest-source",
    "node_id": "quickwit-indexer-6",
    "num_docs": 27112,
    "uncompressed_docs_size_in_bytes": 81336549,
    "time_range": {
      "start": 1755676200,
      "end": 1755676229
    },
    "create_timestamp": 1755676231,
    "maturity": {
      "type": "immature",
      "maturation_period_millis": 172800000
    },
    "tags": [
      "app_name!",
      "app_name:spot-trade-etl"
    ],
    "footer_offsets": {
      "start": 12433661,
      "end": 12444731
    },
    "delete_opstamp": 0,
    "num_merge_ops": 0,
    "doc_mapping_uid": "01JZS10GJPZMKRNWPP4YZBPRY5"
  },
  {
    "split_state": "Published",
    "update_timestamp": 1755212085,
    "publish_timestamp": 1755212085,
    "version": "0.9",
    "split_id": "01K2NCBQTMYYTRC03WAWHYRW7W",
    "index_uid": "spot-trade-etl_exlogs:01JZS10GJQXVHDP72FC2XJN54T",
    "partition_id": 0,
    "source_id": "_ingest-source",
    "node_id": "quickwit-indexer-0",
    "num_docs": 15418795,
    "uncompressed_docs_size_in_bytes": 46302172094,
    "time_range": {
      "start": 1755208585,
      "end": 1755211654
    },
    "create_timestamp": 1755211966,
    "maturity": {
      "type": "mature"
    },
    "tags": [
      "app_name!",
      "app_name:spot-trade-etl"
    ],
    "footer_offsets": {
      "start": 6877313999,
      "end": 6878343730
    },
    "delete_opstamp": 0,
    "num_merge_ops": 3,
    "doc_mapping_uid": "01JZS10GJPZMKRNWPP4YZBPRY5"
  },
  {
    "split_state": "Published",
    "update_timestamp": 1755370273,
    "publish_timestamp": 1755370273,
    "version": "0.9",
    "split_id": "01K2T39JXKAJDVW73HFZ16M43T",
    "index_uid": "spot-trade-etl_exlogs:01JZS10GJQXVHDP72FC2XJN54T",
    "partition_id": 0,
    "source_id": "_ingest-source",
    "node_id": "quickwit-indexer-3",
    "num_docs": 11589080,
    "uncompressed_docs_size_in_bytes": 34771352380,
    "time_range": {
      "start": 1755367830,
      "end": 1755369891
    },
    "create_timestamp": 1755370173,
    "maturity": {
      "type": "mature"
    },
    "tags": [
      "app_name!",
      "app_name:spot-trade-etl"
    ],
    "footer_offsets": {
      "start": 5154319706,
      "end": 5155091247
    },
    "delete_opstamp": 0,
    "num_merge_ops": 3,
    "doc_mapping_uid": "01JZS10GJPZMKRNWPP4YZBPRY5"
  },

Expected value:

"tags": [
      "app_name!",
      "app_name:spot-trade-etl"
    ],

Get value:

"tags": [
      "app_name!",
      "app_name:isolatedbalanceservice",
      "app_name:kafkaservice",
      "app_name:orderkafkacounterconsumer",
      "app_name:publishadaptor",
      "app_name:spotbalanceservice",
      "app_name:tradedetailservice"
    ],
  1. Output of quickwit --version
Quickwit 0.8.0 (x86_64-unknown-linux-gnu 2025-04-08T17:10:33Z 206ebf7)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report provides no source file or test entry point. Start by reproducing the issue with the supplied Quickwit 0.8.0 configuration, comparing tag_fields output before and after adding uri to field_mappings; done means app_name tags remain unchanged after schema changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.