manticoresoftware / manticoresoftware/manticoresearch-python-asyncio
SQL response model drops the `scroll` token, making scroll pagination unusable via UtilsApi.sql()
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
The daemon returns a `scroll` token in the `/sql` response, but the client's SQL response
model does not declare that field, so it is discarded during deserialization and never
reaches the caller.
A scroll continuation requires passing the previous token back
(`... OPTION scroll=''`), so with no way to read it, **scroll pagination cannot be
used through `UtilsApi.sql()` at all**.
The `/search` path is unaffected — `SearchResponse` does declare `scroll`.
## Environment
- Client: `manticoresearch-asyncio`
2.0.0 / 2.1.0 — `sql_obj_response.py` is identical in both and neither declares `scroll`.
- Daemon: Manticore 28.6.6
- Python 3.12
## Setup
```sql
DROP TABLE IF EXISTS scroll_bug;
CREATE TABLE scroll_bug (title text, color string);
INSERT INTO scroll_bug (id, title, color) VALUES
(1,'document number 1 about widgets','c1'),
(2,'document number 2 about widgets','c2'),
(3,'document number 3 about widgets','c0'),
(4,'document number 4 about widgets','c1'),
(5,'document number 5 about widgets','c2'),
(6,'document number 6 about widgets','c0'),
(7,'document number 7 about widgets','c1'),
(8,'document number 8 about widgets','c2'),
(9,'document number 9 about widgets','c0'),
(10,'document number 10 about widgets','c1');
```
## Reproduction
```python
import asyncio, json
from manticoresearch import ApiClient, Configuration, UtilsApi
HOST = "http://127.0.0.1:9308"
SQL = ("SELECT id AS docid, title FROM scroll_bug WHERE MATCH('widgets') "
"ORDER BY weight() DESC, id ASC LIMIT 3")
async def main():
client = ApiClient(Configuration(host=HOST))
# what the daemon sends
r = await client.rest_client.request(
'POST', f'{HOST}/sql',
headers={'Content-Type': 'application/x-www-form-urlencoded'},
post_params=[('query', SQL)])
print("wire :", list(json.loads(await r.read()).keys()))
# what the client hands back
resp = await UtilsApi(client).sql(SQL, raw_response=False)
print("client:", list(resp.to_dict().keys()))
await client.close()
asyncio.run(main())
```
Output:
```
wire : ['took', 'timed_out', 'hits', 'scroll']
client: ['hits', 'took', 'timed_out']
```
`raw_response=True` does not help either — it returns the columns/data form, also with no
`scroll`.
Same thing with `curl`, confirming the token is genuinely on the wire:
```bash
curl -s -X POST http://127.0.0.1:9308/sql --data-urlencode \
"query=SELECT id AS docid, title FROM scroll_bug WHERE MATCH('widgets') ORDER BY weight() DESC, id ASC LIMIT 3" \
| jq 'keys'
# ["hits","scroll","timed_out","took"]
```
## Root cause
`SqlObjResponse` declares only three properties
(`manticoresearch/models/sql_obj_response.py`, line 33):
```python
__properties: ClassVar[List[str]] = ["hits", "took", "timed_out"]
```
`SearchResponse` (`manticoresearch/models/search_response.py`, line 44) does carry it:
```python
__properties: ClassVar[List[str]] = ["took", "timed_out", "aggregations", "hits",
"profile", "scroll", "warning", ...]
```
`UtilsApi.sql()` maps its `200` response to `SqlResponse`
(`manticoresearch/api/utils_api.py`, line 101), whose `oneOf` member for the hits form is
`SqlObjResponse` — so `scroll` is dropped on the way through.
Confirmed at runtime:
```python
from manticoresearch.models.sql_obj_response import SqlObjResponse
from manticoresearch.models.search_response import SearchResponse
'scroll' in SqlObjResponse.model_fields # False
'scroll' in SearchResponse.model_fields # True
```
## Impact
Scroll is documented as the way to page beyond `max_matches`
(https://manual.manticoresearch.com/Searching/Pagination), but on the SQL path it is
currently unreachable through the client:
- the initial request's token is discarded, so there is nothing to pass to
`OPTION scroll='...'`
- `SHOW SCROLL` is not an alternative over HTTP, since each request is a separate session
and it returns an empty set
The mechanism itself works correctly once the token is readable. Reading the raw response
instead of the model, scrolling the 10-row table above in pages of 3:
```
pages=4 rows=10 unique=10 dupes=0
```
## Suggested fix
Add `scroll` to the SQL response schema so it matches `SearchResponse`:
```python
# manticoresearch/models/sql_obj_response.py
scroll: Optional[StrictStr] = Field(default=None, description="Scroll token to be used for pagination")
__properties: ClassVar[List[str]] = ["hits", "took", "timed_out", "scroll"]
```
Presumably this belongs in the OpenAPI spec's SQL response definition rather than in the
generated file directly.
## Workaround
The token can be read by going around the model and using the client's own transport, so
no extra dependency is required:
```python
r = await client.rest_client.request(
'POST', f'{HOST}/sql',
headers={'Content-Type': 'application/x-www-form-urlencoded'},
post_params=[('query', sql)])
body = json.loads(await r.read())
token = body.get('scroll')
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the SQL response definition and manticoresearch/models/sql_obj_response.py, then trace the 200 response mapping in manticoresearch/api/utils_api.py. Verify the model includes the scroll field and that UtilsApi.sql() exposes it, using the supplied SQL reproduction to confirm the token survives deserialization and supports continuation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100