quickwit-oss / quickwit-oss/quickwit
Adding new fields to doc_mapping unexpectedly alters tag_fields behavior
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Describe the bug
When adding new fields to the doc_mapping configuration, it causes unexpected changes in the values retrieved from fields defined in tag_fields.
For example:
In the current index configuration (version 0.9, index_id: spot-trade-etl_exlogs), we have tag_fields: [app_name]properly defined. However, after adding the 'uri' field to field_mappings, the way tag content is processed changes unexpectedly.
Current Behavior
Modifying doc_mapping (such as adding the uri field) inadvertently affects the retrieval mechanism of fields marked in tag_fields (app_name in this case), producing unexpected values.
Expected behavior
- tag_fields values should remain stable
- Schema modifications (add/remove fields) should not impact existing tag_fields
Configuration:
version: 0.9
index_id: spot-trade-etl_exlogs
doc_mapping:
mode: dynamic
dynamic_mapping:
field_mappings:
- name: "@timestamp"
type: datetime
input_formats:
- iso8601
- "%Y-%m-%d %H:%M:%S.%f"
output_format: "%Y-%m-%d %H:%M:%S.%f"
fast: true
fast_precision: microseconds
- name: time
type: datetime
input_formats:
- "%Y-%m-%d %H:%M:%S.%f"
- iso8601
output_format: "%Y-%m-%d %H:%M:%S.%f"
fast: true
fast_precision: milliseconds
- name: level
type: text
fast: true
- name: msg
type: text
tokenizer: chinese_compatible
record: position
fieldnorms: true
fast:
normalizer: lowercase
- name: log
type: text
tokenizer: chinese_compatible
record: position
fieldnorms: true
fast:
normalizer: lowercase
- name: tid
type: text
tokenizer: raw
fast: true
stored: true
record: basic
- name: biz_monitor_ctx
type: text
record: position
- name: uri
type: text
record: position
- name: event
type: text
fast: true
record: position
- name: app_name
type: text
tokenizer: raw
fast: true
- name: namespace
type: text
fast: true
tokenizer: raw
- name: pod_name
type: text
fast: true
record: position
- name: node_name
type: text
fast: true
tokenizer: raw
timestamp_field: "@timestamp"
tag_fields: [app_name]
search_settings:
default_search_fields: [app_name, pod_name]
indexing_settings:
commit_timeout_secs: 30
merge_policy:
type: "stable_log"
min_level_num_docs: 100000
merge_factor: 10
max_merge_factor: 12
retention:
period: 15 day
schedule: hourly
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1755676264,
"publish_timestamp": 1755676264,
"version": "0.9",
"split_id": "01K3379N8EMHT3MYJ4WD9VVFQT",
"index_uid": "spot-trade-etl_exlogs:01JZS10GJQXVHDP72FC2XJN54T",
"partition_id": 0,
"source_id": "_ingest-source",
"node_id": "quickwit-indexer-1",
"num_docs": 37103,
"uncompressed_docs_size_in_bytes": 111738688,
"time_range": {
"start": 1755676201,
"end": 1755676231
},
"create_timestamp": 1755676232,
"maturity": {
"type": "immature",
"maturation_period_millis": 172800000
},
"tags": [
"app_name!",
"app_name:spot-trade-etl"
],
"footer_offsets": {
"start": 17010937,
"end": 17023033
},
"delete_opstamp": 0,
"num_merge_ops": 0,
"doc_mapping_uid": "01JZS10GJPZMKRNWPP4YZBPRY5"
},
{
"split_state": "Published",
"update_timestamp": 1755676840,
"publish_timestamp": 1755676840,
"version": "0.9",
"split_id": "01K337WZ9CF81Z48RG0FPXTY5B",
"index_uid": "spot-trade-etl_exlogs:01JZS10GJQXVHDP72FC2XJN54T",
"partition_id": 0,
"source_id": "_ingest-source",
"node_id": "quickwit-indexer-7",
"num_docs": 377859,
"uncompressed_docs_size_in_bytes": 1127572890,
"time_range": {
"start": 1755676750,
"end": 1755676834
},
"create_timestamp": 1755676839,
"maturity": {
"type": "immature",
"maturation_period_millis": 172800000
},
"tags": [
"app_name!",
"app_name:isolatedbalanceservice",
"app_name:kafkaservice",
"app_name:orderkafkacounterconsumer",
"app_name:publishadaptor",
"app_name:spotbalanceservice",
"app_name:tradedetailservice"
],
"footer_offsets": {
"start": 169363916,
"end": 169403055
},
"delete_opstamp": 0,
"num_merge_ops": 1,
"doc_mapping_uid": "01K337V0YBCQX3B0BDHGZBBBFM"
},
{
"split_state": "MarkedForDeletion",
"update_timestamp": 1755676236,
"publish_timestamp": 1755676236,
"version": "0.9",
"split_id": "01K3379M7NH0WBHF2Z827B7FZY",
"index_uid": "spot-trade-etl_exlogs:01JZS10GJQXVHDP72FC2XJN54T",
"partition_id": 0,
"source_id": "_ingest-source",
"node_id": "quickwit-indexer-6",
"num_docs": 27112,
"uncompressed_docs_size_in_bytes": 81336549,
"time_range": {
"start": 1755676200,
"end": 1755676229
},
"create_timestamp": 1755676231,
"maturity": {
"type": "immature",
"maturation_period_millis": 172800000
},
"tags": [
"app_name!",
"app_name:spot-trade-etl"
],
"footer_offsets": {
"start": 12433661,
"end": 12444731
},
"delete_opstamp": 0,
"num_merge_ops": 0,
"doc_mapping_uid": "01JZS10GJPZMKRNWPP4YZBPRY5"
},
{
"split_state": "Published",
"update_timestamp": 1755212085,
"publish_timestamp": 1755212085,
"version": "0.9",
"split_id": "01K2NCBQTMYYTRC03WAWHYRW7W",
"index_uid": "spot-trade-etl_exlogs:01JZS10GJQXVHDP72FC2XJN54T",
"partition_id": 0,
"source_id": "_ingest-source",
"node_id": "quickwit-indexer-0",
"num_docs": 15418795,
"uncompressed_docs_size_in_bytes": 46302172094,
"time_range": {
"start": 1755208585,
"end": 1755211654
},
"create_timestamp": 1755211966,
"maturity": {
"type": "mature"
},
"tags": [
"app_name!",
"app_name:spot-trade-etl"
],
"footer_offsets": {
"start": 6877313999,
"end": 6878343730
},
"delete_opstamp": 0,
"num_merge_ops": 3,
"doc_mapping_uid": "01JZS10GJPZMKRNWPP4YZBPRY5"
},
{
"split_state": "Published",
"update_timestamp": 1755370273,
"publish_timestamp": 1755370273,
"version": "0.9",
"split_id": "01K2T39JXKAJDVW73HFZ16M43T",
"index_uid": "spot-trade-etl_exlogs:01JZS10GJQXVHDP72FC2XJN54T",
"partition_id": 0,
"source_id": "_ingest-source",
"node_id": "quickwit-indexer-3",
"num_docs": 11589080,
"uncompressed_docs_size_in_bytes": 34771352380,
"time_range": {
"start": 1755367830,
"end": 1755369891
},
"create_timestamp": 1755370173,
"maturity": {
"type": "mature"
},
"tags": [
"app_name!",
"app_name:spot-trade-etl"
],
"footer_offsets": {
"start": 5154319706,
"end": 5155091247
},
"delete_opstamp": 0,
"num_merge_ops": 3,
"doc_mapping_uid": "01JZS10GJPZMKRNWPP4YZBPRY5"
},
Expected value:
"tags": [
"app_name!",
"app_name:spot-trade-etl"
],
Get value:
"tags": [
"app_name!",
"app_name:isolatedbalanceservice",
"app_name:kafkaservice",
"app_name:orderkafkacounterconsumer",
"app_name:publishadaptor",
"app_name:spotbalanceservice",
"app_name:tradedetailservice"
],
- Output of
quickwit --version
Quickwit 0.8.0 (x86_64-unknown-linux-gnu 2025-04-08T17:10:33Z 206ebf7)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The report provides no source file or test entry point. Start by reproducing the issue with the supplied Quickwit 0.8.0 configuration, comparing tag_fields output before and after adding uri to field_mappings; done means app_name tags remain unchanged after schema changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100