[Bug]: convert-workload mistranslates date bounds: a space separator produces HTTP 400, and a whole-day lte loses a day
- Dominant language
- Python
- Stars
- 3
- Forks
- 5
- Avg merge
- 1d 23m
- Merged PRs (30d)
- 5
Description
### Description
`convert-workload` mistranslates two kinds of OpenSearch date bound. Both are
independent of #68 (bracket inclusivity) and both survive the fix in #69, because
they happen earlier, in `_convert_date_to_solr_format`.
**1. A datetime with a space separator is passed through verbatim, and Solr rejects it**
`yyyy-MM-dd HH:mm:ss` is not in the format map, so the value is logged and used
as-is. A space inside a range term ends the term, and Solr answers HTTP 400 —
the converted operation cannot run at all.
```
$ convert-workload # nyc_taxis date_histogram_calendar_interval
Unknown OpenSearch date format: 'yyyy-MM-dd HH:mm:ss'
Could not parse date '2015-01-01 00:00:00', using as-is
-> dropoff_datetime:[2015-01-01 00:00:00 TO 2016-01-01 00:00:00]
```
Against Solr 10.0.0:
```
HTTP 400 org.apache.solr.search.SyntaxError: Cannot parse
'dropoff_datetime:[2015-01-01 00:00:00 TO 2016-01-01 00:00:00]':
Encountered " "00:00:00 "" at line 1, column 28.
```
Two nyc_taxis operations (`date_histogram_calendar_interval`,
`date_histogram_fixed_interval`) are affected.
**2. A whole-day `lte` is not rounded, and loses a day**
A bound with no time names a *day*, and OpenSearch rounds it to that day's edge:
`lte` and `gt` to its LAST millisecond, `gte` and `lt` to its first. Solr rounds
nothing. Only the two that move to the end of the day need translating, and the
converter translates neither.
For nyc_taxis `date_histogram_facet` (`lte: 21/01/2015`, format `dd/MM/yyyy`),
upstream covers the whole of 21 January; the converter cuts the filter at the
first instant of the 21st.
### Measured effect
300,649 nyc_taxis documents indexed into Solr 10.0.0 and OpenSearch 3.8.0 from
one file, with the same ids, ids and contents verified on both sides:
| | OpenSearch | converted | corrected |
|---|---:|---:|---:|
| `date_histogram_facet` matches | 17,651 | 16,811 | 17,651 |
| daily buckets returned | 21 | 20 | 21 |
The id sets are equal after the fix; before it, 840 documents are missing and 0
are extra, and the bucket for `2015-01-21` is absent entirely rather than short.
### Expected
Both bounds should convert so the Solr query selects the same documents as the
OpenSearch query it came from:
```
dropoff_datetime:[2015-01-01T00:00:00Z TO 2016-01-01T00:00:00Z}
dropoff_datetime:[2015-01-01T00:00:00Z TO 2015-01-22T00:00:00Z}
```
Naming the following day's first instant with an exclusive bracket says "the whole
of the 21st" without depending on how fine Solr's date precision happens to be.
### Version
`main` at 6c6c48a, and reproduced with #69 applied.
Contributor guide
Research direction
Start at `_convert_date_to_solr_format`, identified as the earlier conversion point, and reproduce the two affected `convert-workload` nyc_taxis operations: `date_histogram_calendar_interval`, `date_histogram_fixed_interval`, and `date_histogram_facet`. Verify that space-separated datetimes become the expected Solr bounds and that whole-day end bounds cover the complete day, then confirm the converted queries select the same documents and buckets as OpenSearch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100