[Bug]: web read returns only a related-posts article card (~600B) instead of the page body — <article> card list shadows <main> content
- Dominant language
- JavaScript
- Stars
- 29.5k
- Forks
- 2.9k
- Avg merge
- 15h 36m
- Merged PRs (30d)
- 70
Description
## Summary
`opencli web read --url` returns ~600 B of Markdown (title + one related-post card) for a page whose real body is ~19 KB of text inside ``. Reproduced on opencli 1.8.4 and 1.8.7 (latest).
The root cause is in the extraction heuristic (`clis/web/read.js`, `buildRenderAwareExtractorJs`): when the page contains **multiple `` elements that are not the main content** (here: a "related articles" card list), the heuristic picks the largest `` (a 435-char card), and since `textLen(contentEl) >= 200` the `` fallback is never reached.
## Reproduction
```
opencli web read --url "https://quant67.com/post/cilium/15-gateway-boundary/15-gateway-boundary.html"
```
Output (both 1.8.4 and 1.8.7):
```
size: 609.0 B
```
Saved Markdown contains only:
```markdown
# 【Cilium / eBPF】Gateway API / 南北向边界:与本系列东西向焦点的分工
> 原文链接: https://quant67.com/post/cilium/15-gateway-boundary/15-gateway-boundary.html
---
2026-08-19 · kubernetes / network
### [【Envoy Gateway】对照替代路径:Cilium Gateway、Istio、Ingress 与其他实现](https://quant67.com/post/envoy-gateway/14-vs-alternatives/14-vs-alternatives.html)
从翻译内核与排障坐标对照 Envoy Gateway v1.9.0 与 Cilium Gateway、Istio/GAMMA、Ingress 注解模型,以及 Contour / NGINX Gateway Fabric 实现叶;机制差表,不做延迟排行榜。
```
(The "related articles" card that got selected.)
Expected: the full article body — the page's real content lives in `` (~19.6 KB HTML / ~6.6 K chars of plain text, includes `
`/`` blocks). Verified present in server-rendered HTML via plain curl; not a JS-rendering issue.
## Root-cause analysis
In `buildRenderAwareExtractorJs`:
```js
const articles = document.querySelectorAll('article');
if (articles.length === 1) {
contentEl = articles[0];
} else if (articles.length > 1) {
let maxLen = 0;
articles.forEach(a => {
const len = textLen(a);
if (len > maxLen) { maxLen = len; contentEl = a; }
});
}
if (!contentEl) contentEl = document.querySelector('[role="main"]');
if (!contentEl) contentEl = document.querySelector('main');
...
if (!contentEl || textLen(contentEl) < 200) contentEl = document.body;
```
This page contains **4 `` elements**, all related-post cards (~398 / 394 / 435 / 418 chars each). The largest card (435 chars) wins the "pick the longest ``" branch, and because 435 > 200, the `` and `document.body` fallbacks are skipped entirely. The real article is therefore never selected.
The heuristic assumes that when multiple `` elements exist, the largest one is the main content (e.g., an article with a comments section that also uses ``). That assumption breaks for sites that use `` tags for card/list components (related posts, blog cards, etc.) — a pattern that's becoming common.
## Suggested fix (for maintainers)
When multiple ``s exist, compare the candidate against ``, `[role="main"]`, and overall body text length before committing:
```js
// e.g., keep the largest only if it beats the /[role=main]/body
```
or add a sanity check: if the page has a `` whose text length is significantly larger than the best ``, prefer ``.
## Impact
Any site using `` for non-content components (card lists, related posts, teasers) gets silently truncated — no error raised, `status: success`. In our case this silently degraded 57 digested article summaries before we noticed.
## Question: is there an escape hatch?
Is there already a supported way to force the extraction container for an arbitrary URL (e.g. `opencli web read --selector main.post-body`, or a `browser extract`-style selector flag wired into `web read`)? The SKILL docs mention `browser extract --selector ` — can `web read` accept a custom container selector, or should callers fall back to `browser extract`? A `--selector` flag on `web read` would unblock affected sites immediately without waiting for the heuristic fix.
Contributor guide
Research direction
Reproduce the URL with opencli web read, then inspect clis/web/read.js and buildRenderAwareExtractorJs, focusing on how multiple article elements are chosen before main and role="main" fallbacks. Confirm the fix returns the full main.post-body content, including code blocks, without regressing pages where an article is the correct container; the selector escape hatch is an additional scope question.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100