quickwit-oss / quickwit-oss/quickwit
Dynamic mode indexes may have spaces in the names, but strict indexes can't
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
Describe the bug
I try to create two indexes, one like
version: 0.7
index_id: sales-schemaless
doc_mapping:
mode: dynamic
indexing_settings:
commit_timeout_secs: 30
and one like
version: 0.7
index_id: sales-schema
doc_mapping:
mode: strict
timestamp_field: "Order Date"
field_mappings:
- name: Order Date
type: datetime
input_formats:
- iso8601
fast: true
- name: "Ship Date"
type: datetime
input_formats:
- iso8601
fast: true
- name: "Country"
type: text
indexing_settings:
commit_timeout_secs: 30
the second one won't create with the message;
./quickwit index create --index-config sales_schema_config.yaml
❯ Creating index...
✘ command failed: API error: (code=400 Bad Request, message=invalid config: failed to parse YAML file: field name `Order Date` contains illegal characters. field names must only contain uppercase and lowercase ASCII letters, digits, hyphens `-`, periods `.`, and underscores `_`)
Caused by:
(code=400 Bad Request, message=invalid config: failed to parse YAML file: field name `Order Date` contains illegal characters. field names must only contain uppercase and lowercase ASCII letters, digits, hyphens `-`, periods `.`, and underscores `_`)
but the first one will create and actually work with a jsonl that has spaces in the keys! So i can't force a schema on a data set that works if the schema is dynamic.
Steps to reproduce (if applicable)
Steps to reproduce the behavior:
- create the schemas above
2.insert the following data into the dynamic schema
{"Country":"Azerbaijan","Order Date":"2014-10-08T00:00:00+00:00","Ship Date":"2014-10-23T00:00:00+00:00"}
with the command
./quickwit tool local-ingest --index sales-schemaless --input-path
- observe that it works :)
- try to create the second schema
- observe it doesn't work
Expected behavior
I'd expect the strict schemas to be able to emulate the dynamic schemas
Configuration:
Please provide:
- Output of
quickwit --version
Quickwit 0.8.0 (aarch64-unknown-linux-gnu 2024-03-18T15:20:23Z 7af4596)
- The index_config.yaml
see above^^^
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the index_config.yaml examples and reproduce the behavior through quickwit index create, then compare strict field-name validation with dynamic indexing via quickwit tool local-ingest. Done means a strict schema can accept the same space-containing JSONL keys as the dynamic schema, with coverage for the reproduced configurations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100