jackwener / jackwener/OpenCLI

[Bug]: web read returns only a related-posts article card (~600B) instead of the page body — <article> card list shadows <main> content

Open
#2,469 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
29.5k
Forks
2.9k
Avg merge
15h 36m
Merged PRs (30d)
70

Description

## Summary

`opencli web read --url` returns ~600 B of Markdown (title + one related-post card) for a page whose real body is ~19 KB of text inside ``. Reproduced on opencli 1.8.4 and 1.8.7 (latest).

The root cause is in the extraction heuristic (`clis/web/read.js`, `buildRenderAwareExtractorJs`): when the page contains **multiple `` elements that are not the main content** (here: a "related articles" card list), the heuristic picks the largest `` (a 435-char card), and since `textLen(contentEl) >= 200` the `` fallback is never reached.

## Reproduction

```
opencli web read --url "https://quant67.com/post/cilium/15-gateway-boundary/15-gateway-boundary.html"
```

Output (both 1.8.4 and 1.8.7):

```
size: 609.0 B
```

Saved Markdown contains only:

```markdown
# 【Cilium / eBPF】Gateway API / 南北向边界:与本系列东西向焦点的分工
> 原文链接: https://quant67.com/post/cilium/15-gateway-boundary/15-gateway-boundary.html

---

2026-08-19 · kubernetes / network

### [【Envoy Gateway】对照替代路径:Cilium Gateway、Istio、Ingress 与其他实现](https://quant67.com/post/envoy-gateway/14-vs-alternatives/14-vs-alternatives.html)

从翻译内核与排障坐标对照 Envoy Gateway v1.9.0 与 Cilium Gateway、Istio/GAMMA、Ingress 注解模型,以及 Contour / NGINX Gateway Fabric 实现叶;机制差表,不做延迟排行榜。
```

(The "related articles" card that got selected.)

Expected: the full article body — the page's real content lives in `` (~19.6 KB HTML / ~6.6 K chars of plain text, includes `

`/`` blocks). Verified present in server-rendered HTML via plain curl; not a JS-rendering issue.

## Root-cause analysis

In `buildRenderAwareExtractorJs`:

```js
const articles = document.querySelectorAll('article');
if (articles.length === 1) {
contentEl = articles[0];
} else if (articles.length > 1) {
let maxLen = 0;
articles.forEach(a => {
const len = textLen(a);
if (len > maxLen) { maxLen = len; contentEl = a; }
});
}
if (!contentEl) contentEl = document.querySelector('[role="main"]');
if (!contentEl) contentEl = document.querySelector('main');
...
if (!contentEl || textLen(contentEl) < 200) contentEl = document.body;
```

This page contains **4 `` elements**, all related-post cards (~398 / 394 / 435 / 418 chars each). The largest card (435 chars) wins the "pick the longest ``" branch, and because 435 > 200, the `` and `document.body` fallbacks are skipped entirely. The real article is therefore never selected.

The heuristic assumes that when multiple `` elements exist, the largest one is the main content (e.g., an article with a comments section that also uses ``). That assumption breaks for sites that use `` tags for card/list components (related posts, blog cards, etc.) — a pattern that's becoming common.

## Suggested fix (for maintainers)

When multiple ``s exist, compare the candidate against ``, `[role="main"]`, and overall body text length before committing:

```js
// e.g., keep the largest only if it beats the /[role=main]/body
```

or add a sanity check: if the page has a `` whose text length is significantly larger than the best ``, prefer ``.

## Impact

Any site using `` for non-content components (card lists, related posts, teasers) gets silently truncated — no error raised, `status: success`. In our case this silently degraded 57 digested article summaries before we noticed.

## Question: is there an escape hatch?

Is there already a supported way to force the extraction container for an arbitrary URL (e.g. `opencli web read --selector main.post-body`, or a `browser extract`-style selector flag wired into `web read`)? The SKILL docs mention `browser extract --selector ` — can `web read` accept a custom container selector, or should callers fall back to `browser extract`? A `--selector` flag on `web read` would unblock affected sites immediately without waiting for the heuristic fix.

Contributor guide

Open the contributing guide

Research direction

Reproduce the URL with opencli web read, then inspect clis/web/read.js and buildRenderAwareExtractorJs, focusing on how multiple article elements are chosen before main and role="main" fallbacks. Confirm the fix returns the full main.post-body content, including code blocks, without regressing pages where an article is the correct container; the selector escape hatch is an additional scope question.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.