[Bug/Feature] Weibo keyword search only crawls deduplicated results; unable to crawl "all search results" after similar results are omitted

Description

When using the Weibo keyword search crawler, the current implementation appears to crawl only the default search result set returned by the mobile Weibo search API. However, Weibo search may omit similar results by default.

For example, when searching the keyword:

镇干部钓鱼溺亡

the crawler stops at page 17 with the following API response:

{'ok': 0, 'msg': '这里还没有内容', 'data': {'cards': []}}

The crawler then retries several times and finally exits with:

tenacity.RetryError: RetryError[<Future ... raised DataFetchError>]

At first this looks like the crawler has reached the real end of search results. But after manually opening the Weibo web search page in the browser, the bottom of the page shows:

找到166条结果，部分相似结果已省略，您可以点击查看全部搜索结果

This exactly corresponds to around 16 pages of results. In other words, the crawler is currently collecting only the default deduplicated / similar-results-omitted result set, not the complete result set available after clicking "查看全部搜索结果".

Current behavior

The crawler uses the mobile Weibo search API:

https://m.weibo.cn/api/container/getIndex

with params similar to:

containerid=100103type=1&q={keyword}
page_type=searchall
page={page}

When Weibo returns:

{"ok": 0, "msg": "这里还没有内容", "data": {"cards": []}}

the crawler treats it as an error and exits after retrying.

Expected behavior

It would be helpful if the Weibo crawler could support crawling the full search result set, equivalent to manually clicking:

查看全部搜索结果

on the Weibo web search page.

Possible expected behavior:

1. Add a configuration option such as:

WEIBO_SEARCH_FULL_RESULT = True

2. When enabled, use the full-result search mode, probably by adding / supporting the nodup parameter or by using the corresponding web search endpoint.
3. When the search API returns "这里还没有内容", treat it as "no more results" instead of a fatal error, so that the crawler can stop the current keyword gracefully and continue with the next keyword.

Reproduction steps

1. Set platform to Weibo.
2. Use keyword search mode.
3. Set keyword:

KEYWORDS = "镇干部钓鱼溺亡"

4. Set a large enough max crawl count, e.g. more than 200.
5. Run the crawler.
6. The crawler successfully crawls pages 1-16, but page 17 returns "这里还没有内容" and exits with RetryError.
7. Manually opening the Weibo web search page shows that only similar-results-omitted results are displayed by default, and there is an option to view all search results.

Relevant log

[WeiboCrawler.search] search weibo keyword: 镇干部钓鱼溺亡, page: 17

[WeiboClient.request] request GET:
https://m.weibo.cn/api/container/getIndex?containerid=100103type%3D1%26q%3D%E9%95%87%E5%B9%B2%E9%83%A8%E9%92%93%E9%B1%BC%E6%BA%BA%E4%BA%A1&page_type=searchall&page=17
err, res:{'ok': 0, 'msg': '这里还没有内容', 'data': {'cards': []}}

media_platform.weibo.exception.DataFetchError: 这里还没有内容

tenacity.RetryError: RetryError[<Future ... raised DataFetchError>]

Suggested fix

Possible directions:

1. Add a Weibo search config option to enable full search results.
2. Try adding the nodup parameter to Weibo search requests.
3. If the mobile API does not support this parameter reliably, add a fallback mode using the PC web search endpoint, such as s.weibo.com/weibo, and parse result mids before fetching post details through the existing detail method.
4. Gracefully handle "这里还没有内容" as end-of-results instead of throwing a fatal exception.

Environment

• Platform: Weibo
• Crawler type: search
• Python version: 3.11
• Running on Windows
• Login mode: existing browser / CDP mode

Start at the WeiboCrawler.search entry point and the WeiboClient.request path described in the logs; inspect how searchall pages and the "这里还没有内容" response are handled. Compare the existing mobile API request with the suggested nodup or web-search approach, and verify that an exhausted keyword stops cleanly while full-result mode retrieves the expected additional results.