apache / apache/solr-orbit

[Bug]: convert-workload mistranslates date bounds: a space separator produces HTTP 400, and a whole-day lte loses a day

Open
#73 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3
Forks
5
Avg merge
1d 23m
Merged PRs (30d)
5

Description

### Description

`convert-workload` mistranslates two kinds of OpenSearch date bound. Both are
independent of #68 (bracket inclusivity) and both survive the fix in #69, because
they happen earlier, in `_convert_date_to_solr_format`.

**1. A datetime with a space separator is passed through verbatim, and Solr rejects it**

`yyyy-MM-dd HH:mm:ss` is not in the format map, so the value is logged and used
as-is. A space inside a range term ends the term, and Solr answers HTTP 400 —
the converted operation cannot run at all.

```
$ convert-workload # nyc_taxis date_histogram_calendar_interval
Unknown OpenSearch date format: 'yyyy-MM-dd HH:mm:ss'
Could not parse date '2015-01-01 00:00:00', using as-is
-> dropoff_datetime:[2015-01-01 00:00:00 TO 2016-01-01 00:00:00]
```

Against Solr 10.0.0:

```
HTTP 400 org.apache.solr.search.SyntaxError: Cannot parse
'dropoff_datetime:[2015-01-01 00:00:00 TO 2016-01-01 00:00:00]':
Encountered " "00:00:00 "" at line 1, column 28.
```

Two nyc_taxis operations (`date_histogram_calendar_interval`,
`date_histogram_fixed_interval`) are affected.

**2. A whole-day `lte` is not rounded, and loses a day**

A bound with no time names a *day*, and OpenSearch rounds it to that day's edge:
`lte` and `gt` to its LAST millisecond, `gte` and `lt` to its first. Solr rounds
nothing. Only the two that move to the end of the day need translating, and the
converter translates neither.

For nyc_taxis `date_histogram_facet` (`lte: 21/01/2015`, format `dd/MM/yyyy`),
upstream covers the whole of 21 January; the converter cuts the filter at the
first instant of the 21st.

### Measured effect

300,649 nyc_taxis documents indexed into Solr 10.0.0 and OpenSearch 3.8.0 from
one file, with the same ids, ids and contents verified on both sides:

| | OpenSearch | converted | corrected |
|---|---:|---:|---:|
| `date_histogram_facet` matches | 17,651 | 16,811 | 17,651 |
| daily buckets returned | 21 | 20 | 21 |

The id sets are equal after the fix; before it, 840 documents are missing and 0
are extra, and the bucket for `2015-01-21` is absent entirely rather than short.

### Expected

Both bounds should convert so the Solr query selects the same documents as the
OpenSearch query it came from:

```
dropoff_datetime:[2015-01-01T00:00:00Z TO 2016-01-01T00:00:00Z}
dropoff_datetime:[2015-01-01T00:00:00Z TO 2015-01-22T00:00:00Z}
```

Naming the following day's first instant with an exclusive bracket says "the whole
of the 21st" without depending on how fine Solr's date precision happens to be.

### Version

`main` at 6c6c48a, and reproduced with #69 applied.

Contributor guide

Open the contributing guide

Research direction

Start at `_convert_date_to_solr_format`, identified as the earlier conversion point, and reproduce the two affected `convert-workload` nyc_taxis operations: `date_histogram_calendar_interval`, `date_histogram_fixed_interval`, and `date_histogram_facet`. Verify that space-separated datetimes become the expected Solr bounds and that whole-day end bounds cover the complete day, then confirm the converted queries select the same documents and buckets as OpenSearch.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.