manticoresoftware / manticoresoftware/manticoresearch-python-asyncio

SQL response model drops the `scroll` token, making scroll pagination unusable via UtilsApi.sql()

Aperta Adatta ai principianti
#2 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Lingua principale
Python
Stelle
8
Fork
2
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Summary

The daemon returns a `scroll` token in the `/sql` response, but the client's SQL response
model does not declare that field, so it is discarded during deserialization and never
reaches the caller.

A scroll continuation requires passing the previous token back
(`... OPTION scroll=''`), so with no way to read it, **scroll pagination cannot be
used through `UtilsApi.sql()` at all**.

The `/search` path is unaffected — `SearchResponse` does declare `scroll`.

## Environment

- Client: `manticoresearch-asyncio`
2.0.0 / 2.1.0 — `sql_obj_response.py` is identical in both and neither declares `scroll`.
- Daemon: Manticore 28.6.6
- Python 3.12

## Setup

```sql
DROP TABLE IF EXISTS scroll_bug;
CREATE TABLE scroll_bug (title text, color string);
INSERT INTO scroll_bug (id, title, color) VALUES
(1,'document number 1 about widgets','c1'),
(2,'document number 2 about widgets','c2'),
(3,'document number 3 about widgets','c0'),
(4,'document number 4 about widgets','c1'),
(5,'document number 5 about widgets','c2'),
(6,'document number 6 about widgets','c0'),
(7,'document number 7 about widgets','c1'),
(8,'document number 8 about widgets','c2'),
(9,'document number 9 about widgets','c0'),
(10,'document number 10 about widgets','c1');
```

## Reproduction

```python
import asyncio, json
from manticoresearch import ApiClient, Configuration, UtilsApi

HOST = "http://127.0.0.1:9308"
SQL = ("SELECT id AS docid, title FROM scroll_bug WHERE MATCH('widgets') "
"ORDER BY weight() DESC, id ASC LIMIT 3")

async def main():
client = ApiClient(Configuration(host=HOST))

# what the daemon sends
r = await client.rest_client.request(
'POST', f'{HOST}/sql',
headers={'Content-Type': 'application/x-www-form-urlencoded'},
post_params=[('query', SQL)])
print("wire :", list(json.loads(await r.read()).keys()))

# what the client hands back
resp = await UtilsApi(client).sql(SQL, raw_response=False)
print("client:", list(resp.to_dict().keys()))

await client.close()

asyncio.run(main())
```

Output:

```
wire : ['took', 'timed_out', 'hits', 'scroll']
client: ['hits', 'took', 'timed_out']
```

`raw_response=True` does not help either — it returns the columns/data form, also with no
`scroll`.

Same thing with `curl`, confirming the token is genuinely on the wire:

```bash
curl -s -X POST http://127.0.0.1:9308/sql --data-urlencode \
"query=SELECT id AS docid, title FROM scroll_bug WHERE MATCH('widgets') ORDER BY weight() DESC, id ASC LIMIT 3" \
| jq 'keys'
# ["hits","scroll","timed_out","took"]
```

## Root cause

`SqlObjResponse` declares only three properties
(`manticoresearch/models/sql_obj_response.py`, line 33):

```python
__properties: ClassVar[List[str]] = ["hits", "took", "timed_out"]
```

`SearchResponse` (`manticoresearch/models/search_response.py`, line 44) does carry it:

```python
__properties: ClassVar[List[str]] = ["took", "timed_out", "aggregations", "hits",
"profile", "scroll", "warning", ...]
```

`UtilsApi.sql()` maps its `200` response to `SqlResponse`
(`manticoresearch/api/utils_api.py`, line 101), whose `oneOf` member for the hits form is
`SqlObjResponse` — so `scroll` is dropped on the way through.

Confirmed at runtime:

```python
from manticoresearch.models.sql_obj_response import SqlObjResponse
from manticoresearch.models.search_response import SearchResponse
'scroll' in SqlObjResponse.model_fields # False
'scroll' in SearchResponse.model_fields # True
```

## Impact

Scroll is documented as the way to page beyond `max_matches`
(https://manual.manticoresearch.com/Searching/Pagination), but on the SQL path it is
currently unreachable through the client:

- the initial request's token is discarded, so there is nothing to pass to
`OPTION scroll='...'`
- `SHOW SCROLL` is not an alternative over HTTP, since each request is a separate session
and it returns an empty set

The mechanism itself works correctly once the token is readable. Reading the raw response
instead of the model, scrolling the 10-row table above in pages of 3:

```
pages=4 rows=10 unique=10 dupes=0
```

## Suggested fix

Add `scroll` to the SQL response schema so it matches `SearchResponse`:

```python
# manticoresearch/models/sql_obj_response.py
scroll: Optional[StrictStr] = Field(default=None, description="Scroll token to be used for pagination")
__properties: ClassVar[List[str]] = ["hits", "took", "timed_out", "scroll"]
```

Presumably this belongs in the OpenAPI spec's SQL response definition rather than in the
generated file directly.

## Workaround

The token can be read by going around the model and using the client's own transport, so
no extra dependency is required:

```python
r = await client.rest_client.request(
'POST', f'{HOST}/sql',
headers={'Content-Type': 'application/x-www-form-urlencoded'},
post_params=[('query', sql)])
body = json.loads(await r.read())
token = body.get('scroll')
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia con la definizione della risposta SQL e manticoresearch/models/sql_obj_response.py, quindi segui la mappatura della risposta 200 in manticoresearch/api/utils_api.py. Verifica che il modello includa il campo scroll e che UtilsApi.sql() lo esponga, utilizzando la riproduzione SQL fornita per confermare che il token sopravviva alla deserializzazione e supporti la continuazione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
api
Tipo di issue
Bug
Difficoltà
2/5
Tempo stimato
1-3 ore
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
78/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.