manticoresoftware / manticoresoftware/manticoresearch-python-asyncio

SQL response model drops the `scroll` token, making scroll pagination unusable via UtilsApi.sql()

Offen Anfängerfreundlich
#2 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

Vorherrschende Sprache
Python
Sterne
8
Forks
2
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Summary

The daemon returns a `scroll` token in the `/sql` response, but the client's SQL response
model does not declare that field, so it is discarded during deserialization and never
reaches the caller.

A scroll continuation requires passing the previous token back
(`... OPTION scroll=''`), so with no way to read it, **scroll pagination cannot be
used through `UtilsApi.sql()` at all**.

The `/search` path is unaffected — `SearchResponse` does declare `scroll`.

## Environment

- Client: `manticoresearch-asyncio`
2.0.0 / 2.1.0 — `sql_obj_response.py` is identical in both and neither declares `scroll`.
- Daemon: Manticore 28.6.6
- Python 3.12

## Setup

```sql
DROP TABLE IF EXISTS scroll_bug;
CREATE TABLE scroll_bug (title text, color string);
INSERT INTO scroll_bug (id, title, color) VALUES
(1,'document number 1 about widgets','c1'),
(2,'document number 2 about widgets','c2'),
(3,'document number 3 about widgets','c0'),
(4,'document number 4 about widgets','c1'),
(5,'document number 5 about widgets','c2'),
(6,'document number 6 about widgets','c0'),
(7,'document number 7 about widgets','c1'),
(8,'document number 8 about widgets','c2'),
(9,'document number 9 about widgets','c0'),
(10,'document number 10 about widgets','c1');
```

## Reproduction

```python
import asyncio, json
from manticoresearch import ApiClient, Configuration, UtilsApi

HOST = "http://127.0.0.1:9308"
SQL = ("SELECT id AS docid, title FROM scroll_bug WHERE MATCH('widgets') "
"ORDER BY weight() DESC, id ASC LIMIT 3")

async def main():
client = ApiClient(Configuration(host=HOST))

# what the daemon sends
r = await client.rest_client.request(
'POST', f'{HOST}/sql',
headers={'Content-Type': 'application/x-www-form-urlencoded'},
post_params=[('query', SQL)])
print("wire :", list(json.loads(await r.read()).keys()))

# what the client hands back
resp = await UtilsApi(client).sql(SQL, raw_response=False)
print("client:", list(resp.to_dict().keys()))

await client.close()

asyncio.run(main())
```

Output:

```
wire : ['took', 'timed_out', 'hits', 'scroll']
client: ['hits', 'took', 'timed_out']
```

`raw_response=True` does not help either — it returns the columns/data form, also with no
`scroll`.

Same thing with `curl`, confirming the token is genuinely on the wire:

```bash
curl -s -X POST http://127.0.0.1:9308/sql --data-urlencode \
"query=SELECT id AS docid, title FROM scroll_bug WHERE MATCH('widgets') ORDER BY weight() DESC, id ASC LIMIT 3" \
| jq 'keys'
# ["hits","scroll","timed_out","took"]
```

## Root cause

`SqlObjResponse` declares only three properties
(`manticoresearch/models/sql_obj_response.py`, line 33):

```python
__properties: ClassVar[List[str]] = ["hits", "took", "timed_out"]
```

`SearchResponse` (`manticoresearch/models/search_response.py`, line 44) does carry it:

```python
__properties: ClassVar[List[str]] = ["took", "timed_out", "aggregations", "hits",
"profile", "scroll", "warning", ...]
```

`UtilsApi.sql()` maps its `200` response to `SqlResponse`
(`manticoresearch/api/utils_api.py`, line 101), whose `oneOf` member for the hits form is
`SqlObjResponse` — so `scroll` is dropped on the way through.

Confirmed at runtime:

```python
from manticoresearch.models.sql_obj_response import SqlObjResponse
from manticoresearch.models.search_response import SearchResponse
'scroll' in SqlObjResponse.model_fields # False
'scroll' in SearchResponse.model_fields # True
```

## Impact

Scroll is documented as the way to page beyond `max_matches`
(https://manual.manticoresearch.com/Searching/Pagination), but on the SQL path it is
currently unreachable through the client:

- the initial request's token is discarded, so there is nothing to pass to
`OPTION scroll='...'`
- `SHOW SCROLL` is not an alternative over HTTP, since each request is a separate session
and it returns an empty set

The mechanism itself works correctly once the token is readable. Reading the raw response
instead of the model, scrolling the 10-row table above in pages of 3:

```
pages=4 rows=10 unique=10 dupes=0
```

## Suggested fix

Add `scroll` to the SQL response schema so it matches `SearchResponse`:

```python
# manticoresearch/models/sql_obj_response.py
scroll: Optional[StrictStr] = Field(default=None, description="Scroll token to be used for pagination")
__properties: ClassVar[List[str]] = ["hits", "took", "timed_out", "scroll"]
```

Presumably this belongs in the OpenAPI spec's SQL response definition rather than in the
generated file directly.

## Workaround

The token can be read by going around the model and using the client's own transport, so
no extra dependency is required:

```python
r = await client.rest_client.request(
'POST', f'{HOST}/sql',
headers={'Content-Type': 'application/x-www-form-urlencoded'},
post_params=[('query', sql)])
body = json.loads(await r.read())
token = body.get('scroll')
```

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Erste Schritte

  1. Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
  3. Forke das Repository und arbeite in einem Branch.
  4. Öffne einen Pull Request, der die Issue-Nummer nennt.

Rechercherichtung

Beginne mit der SQL-Antwortdefinition und manticoresearch/models/sql_obj_response.py und verfolge dann die Zuordnung der 200-Antwort in manticoresearch/api/utils_api.py. Überprüfe, dass das Modell das Feld scroll enthält und dass UtilsApi.sql() es bereitstellt, und verwende die bereitgestellte SQL-Reproduktion, um zu bestätigen, dass das Token die Deserialisierung übersteht und die Fortsetzung unterstützt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
api
Issue-Typ
Bug
Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Aktivitätsstatus
Ruhig
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
78/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.