manticoresoftware / manticoresoftware/manticoresearch-python-asyncio
SQL response model drops the `scroll` token, making scroll pagination unusable via UtilsApi.sql()
Nessuno ha ancora preso questa issue.
- Lingua principale
- Python
- Stelle
- 8
- Fork
- 2
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Summary
The daemon returns a `scroll` token in the `/sql` response, but the client's SQL response
model does not declare that field, so it is discarded during deserialization and never
reaches the caller.
A scroll continuation requires passing the previous token back
(`... OPTION scroll=''`), so with no way to read it, **scroll pagination cannot be
used through `UtilsApi.sql()` at all**.
The `/search` path is unaffected — `SearchResponse` does declare `scroll`.
## Environment
- Client: `manticoresearch-asyncio`
2.0.0 / 2.1.0 — `sql_obj_response.py` is identical in both and neither declares `scroll`.
- Daemon: Manticore 28.6.6
- Python 3.12
## Setup
```sql
DROP TABLE IF EXISTS scroll_bug;
CREATE TABLE scroll_bug (title text, color string);
INSERT INTO scroll_bug (id, title, color) VALUES
(1,'document number 1 about widgets','c1'),
(2,'document number 2 about widgets','c2'),
(3,'document number 3 about widgets','c0'),
(4,'document number 4 about widgets','c1'),
(5,'document number 5 about widgets','c2'),
(6,'document number 6 about widgets','c0'),
(7,'document number 7 about widgets','c1'),
(8,'document number 8 about widgets','c2'),
(9,'document number 9 about widgets','c0'),
(10,'document number 10 about widgets','c1');
```
## Reproduction
```python
import asyncio, json
from manticoresearch import ApiClient, Configuration, UtilsApi
HOST = "http://127.0.0.1:9308"
SQL = ("SELECT id AS docid, title FROM scroll_bug WHERE MATCH('widgets') "
"ORDER BY weight() DESC, id ASC LIMIT 3")
async def main():
client = ApiClient(Configuration(host=HOST))
# what the daemon sends
r = await client.rest_client.request(
'POST', f'{HOST}/sql',
headers={'Content-Type': 'application/x-www-form-urlencoded'},
post_params=[('query', SQL)])
print("wire :", list(json.loads(await r.read()).keys()))
# what the client hands back
resp = await UtilsApi(client).sql(SQL, raw_response=False)
print("client:", list(resp.to_dict().keys()))
await client.close()
asyncio.run(main())
```
Output:
```
wire : ['took', 'timed_out', 'hits', 'scroll']
client: ['hits', 'took', 'timed_out']
```
`raw_response=True` does not help either — it returns the columns/data form, also with no
`scroll`.
Same thing with `curl`, confirming the token is genuinely on the wire:
```bash
curl -s -X POST http://127.0.0.1:9308/sql --data-urlencode \
"query=SELECT id AS docid, title FROM scroll_bug WHERE MATCH('widgets') ORDER BY weight() DESC, id ASC LIMIT 3" \
| jq 'keys'
# ["hits","scroll","timed_out","took"]
```
## Root cause
`SqlObjResponse` declares only three properties
(`manticoresearch/models/sql_obj_response.py`, line 33):
```python
__properties: ClassVar[List[str]] = ["hits", "took", "timed_out"]
```
`SearchResponse` (`manticoresearch/models/search_response.py`, line 44) does carry it:
```python
__properties: ClassVar[List[str]] = ["took", "timed_out", "aggregations", "hits",
"profile", "scroll", "warning", ...]
```
`UtilsApi.sql()` maps its `200` response to `SqlResponse`
(`manticoresearch/api/utils_api.py`, line 101), whose `oneOf` member for the hits form is
`SqlObjResponse` — so `scroll` is dropped on the way through.
Confirmed at runtime:
```python
from manticoresearch.models.sql_obj_response import SqlObjResponse
from manticoresearch.models.search_response import SearchResponse
'scroll' in SqlObjResponse.model_fields # False
'scroll' in SearchResponse.model_fields # True
```
## Impact
Scroll is documented as the way to page beyond `max_matches`
(https://manual.manticoresearch.com/Searching/Pagination), but on the SQL path it is
currently unreachable through the client:
- the initial request's token is discarded, so there is nothing to pass to
`OPTION scroll='...'`
- `SHOW SCROLL` is not an alternative over HTTP, since each request is a separate session
and it returns an empty set
The mechanism itself works correctly once the token is readable. Reading the raw response
instead of the model, scrolling the 10-row table above in pages of 3:
```
pages=4 rows=10 unique=10 dupes=0
```
## Suggested fix
Add `scroll` to the SQL response schema so it matches `SearchResponse`:
```python
# manticoresearch/models/sql_obj_response.py
scroll: Optional[StrictStr] = Field(default=None, description="Scroll token to be used for pagination")
__properties: ClassVar[List[str]] = ["hits", "took", "timed_out", "scroll"]
```
Presumably this belongs in the OpenAPI spec's SQL response definition rather than in the
generated file directly.
## Workaround
The token can be read by going around the model and using the client's own transport, so
no extra dependency is required:
```python
r = await client.rest_client.request(
'POST', f'{HOST}/sql',
headers={'Content-Type': 'application/x-www-form-urlencoded'},
post_params=[('query', sql)])
body = json.loads(await r.read())
token = body.get('scroll')
```
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Direzione di ricerca
Inizia con la definizione della risposta SQL e manticoresearch/models/sql_obj_response.py, quindi segui la mappatura della risposta 200 in manticoresearch/api/utils_api.py. Verifica che il modello includa il campo scroll e che UtilsApi.sql() lo esponga, utilizzando la riproduzione SQL fornita per confermare che il token sopravviva alla deserializzazione e supporti la continuazione.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- api
- Tipo di issue
- Bug
- Difficoltà
- 2/5
- Tempo stimato
- 1-3 ore
- Stato di attività
- Tranquilla
- Chiarezza
- Specificata chiaramente
- Idoneità per principianti
- 78/100