manticoresoftware / manticoresoftware/manticoresearch-python-asyncio

SQL response model drops the `scroll` token, making scroll pagination unusable via UtilsApi.sql()

Open Beginner friendly
#2 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Summary

The daemon returns a `scroll` token in the `/sql` response, but the client's SQL response
model does not declare that field, so it is discarded during deserialization and never
reaches the caller.

A scroll continuation requires passing the previous token back
(`... OPTION scroll=''`), so with no way to read it, **scroll pagination cannot be
used through `UtilsApi.sql()` at all**.

The `/search` path is unaffected — `SearchResponse` does declare `scroll`.

## Environment

- Client: `manticoresearch-asyncio`
2.0.0 / 2.1.0 — `sql_obj_response.py` is identical in both and neither declares `scroll`.
- Daemon: Manticore 28.6.6
- Python 3.12

## Setup

```sql
DROP TABLE IF EXISTS scroll_bug;
CREATE TABLE scroll_bug (title text, color string);
INSERT INTO scroll_bug (id, title, color) VALUES
(1,'document number 1 about widgets','c1'),
(2,'document number 2 about widgets','c2'),
(3,'document number 3 about widgets','c0'),
(4,'document number 4 about widgets','c1'),
(5,'document number 5 about widgets','c2'),
(6,'document number 6 about widgets','c0'),
(7,'document number 7 about widgets','c1'),
(8,'document number 8 about widgets','c2'),
(9,'document number 9 about widgets','c0'),
(10,'document number 10 about widgets','c1');
```

## Reproduction

```python
import asyncio, json
from manticoresearch import ApiClient, Configuration, UtilsApi

HOST = "http://127.0.0.1:9308"
SQL = ("SELECT id AS docid, title FROM scroll_bug WHERE MATCH('widgets') "
"ORDER BY weight() DESC, id ASC LIMIT 3")

async def main():
client = ApiClient(Configuration(host=HOST))

# what the daemon sends
r = await client.rest_client.request(
'POST', f'{HOST}/sql',
headers={'Content-Type': 'application/x-www-form-urlencoded'},
post_params=[('query', SQL)])
print("wire :", list(json.loads(await r.read()).keys()))

# what the client hands back
resp = await UtilsApi(client).sql(SQL, raw_response=False)
print("client:", list(resp.to_dict().keys()))

await client.close()

asyncio.run(main())
```

Output:

```
wire : ['took', 'timed_out', 'hits', 'scroll']
client: ['hits', 'took', 'timed_out']
```

`raw_response=True` does not help either — it returns the columns/data form, also with no
`scroll`.

Same thing with `curl`, confirming the token is genuinely on the wire:

```bash
curl -s -X POST http://127.0.0.1:9308/sql --data-urlencode \
"query=SELECT id AS docid, title FROM scroll_bug WHERE MATCH('widgets') ORDER BY weight() DESC, id ASC LIMIT 3" \
| jq 'keys'
# ["hits","scroll","timed_out","took"]
```

## Root cause

`SqlObjResponse` declares only three properties
(`manticoresearch/models/sql_obj_response.py`, line 33):

```python
__properties: ClassVar[List[str]] = ["hits", "took", "timed_out"]
```

`SearchResponse` (`manticoresearch/models/search_response.py`, line 44) does carry it:

```python
__properties: ClassVar[List[str]] = ["took", "timed_out", "aggregations", "hits",
"profile", "scroll", "warning", ...]
```

`UtilsApi.sql()` maps its `200` response to `SqlResponse`
(`manticoresearch/api/utils_api.py`, line 101), whose `oneOf` member for the hits form is
`SqlObjResponse` — so `scroll` is dropped on the way through.

Confirmed at runtime:

```python
from manticoresearch.models.sql_obj_response import SqlObjResponse
from manticoresearch.models.search_response import SearchResponse
'scroll' in SqlObjResponse.model_fields # False
'scroll' in SearchResponse.model_fields # True
```

## Impact

Scroll is documented as the way to page beyond `max_matches`
(https://manual.manticoresearch.com/Searching/Pagination), but on the SQL path it is
currently unreachable through the client:

- the initial request's token is discarded, so there is nothing to pass to
`OPTION scroll='...'`
- `SHOW SCROLL` is not an alternative over HTTP, since each request is a separate session
and it returns an empty set

The mechanism itself works correctly once the token is readable. Reading the raw response
instead of the model, scrolling the 10-row table above in pages of 3:

```
pages=4 rows=10 unique=10 dupes=0
```

## Suggested fix

Add `scroll` to the SQL response schema so it matches `SearchResponse`:

```python
# manticoresearch/models/sql_obj_response.py
scroll: Optional[StrictStr] = Field(default=None, description="Scroll token to be used for pagination")
__properties: ClassVar[List[str]] = ["hits", "took", "timed_out", "scroll"]
```

Presumably this belongs in the OpenAPI spec's SQL response definition rather than in the
generated file directly.

## Workaround

The token can be read by going around the model and using the client's own transport, so
no extra dependency is required:

```python
r = await client.rest_client.request(
'POST', f'{HOST}/sql',
headers={'Content-Type': 'application/x-www-form-urlencoded'},
post_params=[('query', sql)])
body = json.loads(await r.read())
token = body.get('scroll')
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the SQL response definition and manticoresearch/models/sql_obj_response.py, then trace the 200 response mapping in manticoresearch/api/utils_api.py. Verify the model includes the scroll field and that UtilsApi.sql() exposes it, using the supplied SQL reproduction to confirm the token survives deserialization and supports continuation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.