quickwit-oss / quickwit-oss/quickwit

Dynamic mode indexes may have spaces in the names, but strict indexes can't

Open
#4,793 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

Describe the bug

I try to create two indexes, one like

version: 0.7
index_id: sales-schemaless
doc_mapping:
  mode: dynamic
indexing_settings:
  commit_timeout_secs: 30

and one like

version: 0.7
index_id: sales-schema
doc_mapping:
  mode: strict
  timestamp_field: "Order Date"
  field_mappings:
    - name: Order Date
      type: datetime
      input_formats:
        - iso8601
      fast: true
    - name: "Ship Date"
      type: datetime
      input_formats:
        - iso8601
      fast: true
    - name: "Country"
      type: text
indexing_settings:
 commit_timeout_secs: 30

the second one won't create with the message;

./quickwit index create --index-config sales_schema_config.yaml
❯ Creating index...
✘ command failed: API error: (code=400 Bad Request, message=invalid config: failed to parse YAML file: field name `Order Date` contains illegal characters. field names must only contain uppercase and lowercase ASCII letters, digits, hyphens `-`, periods `.`, and underscores `_`)

Caused by:
    (code=400 Bad Request, message=invalid config: failed to parse YAML file: field name `Order Date` contains illegal characters. field names must only contain uppercase and lowercase ASCII letters, digits, hyphens `-`, periods `.`, and underscores `_`)

but the first one will create and actually work with a jsonl that has spaces in the keys! So i can't force a schema on a data set that works if the schema is dynamic.

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

  1. create the schemas above
    2.insert the following data into the dynamic schema
{"Country":"Azerbaijan","Order Date":"2014-10-08T00:00:00+00:00","Ship Date":"2014-10-23T00:00:00+00:00"}

with the command

./quickwit tool local-ingest --index sales-schemaless --input-path
  1. observe that it works :)
  2. try to create the second schema
  3. observe it doesn't work

Expected behavior

I'd expect the strict schemas to be able to emulate the dynamic schemas

Configuration:
Please provide:

  1. Output of quickwit --version

Quickwit 0.8.0 (aarch64-unknown-linux-gnu 2024-03-18T15:20:23Z 7af4596)

  1. The index_config.yaml

see above^^^

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the index_config.yaml examples and reproduce the behavior through quickwit index create, then compare strict field-name validation with dynamic indexing via quickwit tool local-ingest. Done means a strict schema can accept the same space-containing JSONL keys as the dynamic schema, with coverage for the reproduced configurations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.