bug(pipelines): docs citations point at unpublished URLs, and the incremental cleaner has diverged

Problem

Two problems in the docs ingestion path, both in logic that kubeflow-pipeline.py and incremental-pipeline.py carry as separate copies.

1. Citation URLs are not the URLs the site publishes

chunk_and_embed builds citation_url by stripping the extension off the content path:

url_path = '/'.join(path_parts[docs_index+1:])
url_path = os.path.splitext(url_path)[0]
citation_url = f"{base_url}/{url_path}"

Hugo publishes a page at its path without the extension and with a trailing slash, and publishes a section's _index.md at the section directory itself. So content/en/docs/components/katib/_index.md lives at /docs/components/katib/, and we cite /docs/components/katib/_index.

kubeflow/website has 220 .md/.html files under content/en/docs; 182 survive the pipeline's len(content) < 50 filter. Requesting the citation URL the pipeline builds for each of those 182, without following redirects:

status	pages
200	0
301	169
404	13

The 169 redirects go to the same URL with a trailing slash and then return 200, so those citations work but are never the canonical URL. The 13 that 404 are all _index.md section landing pages:

• /docs/components/pipelines/operator-guides/installation/_index — the KFP installation guide, 3,614 characters of indexed content
• /docs/components/pipelines/user-guides/components/_index
• /docs/ecosystem/_index, /docs/components/hub/_index, /docs/components/hub/reference/_index
• /docs/components/katib/user-guides/nas/_index, /docs/components/workspaces/operator-guides/_index
• six _index pages under /docs/components/pipelines/legacy-v1/

Since #237 these URLs are the source pills the widget renders under tool-grounded answers, so a wrong one is a dead pill.

2. The incremental pipeline no longer cleans pages the way the full one does

Both components write into the same kubeflow_docs collection. The cleaner in chunk_and_embed was rewritten in #234; chunk_and_embed_incremental still runs the older version.

They no longer agree on which pages exist. Over the same 220 source files the full cleaner keeps 182 and the incremental one keeps 183: it drops components/pipelines/concepts/metadata.md (1,030 characters down to 21, below the 50-character floor) and keeps two ecosystem/ogx/ stubs the full pipeline discards.

Where both keep a page, the text differs. Three causes:

• Markdown links are converted after bare URLs are removed. [web app](https://...) becomes [web app](, and \[([^\]]+)\]\([^\)]+\) then runs to the next ) anywhere in the file, deleting everything in between. This is what the comment added in #234 warns about.
• re.sub(r'\s+', ' ', content) collapses every newline, so YAML and code blocks lose their line structure.
• The frontmatter regex is unanchored (^\s*[+\-]{3,}... with re.MULTILINE), so a --- separator further down the page can match and take the text before the next one with it.

Across the 182 pages the full pipeline indexes, 101 lose text under the incremental cleaner — 181,491 characters, 23.8% of the corpus. ecosystem/kserve/webapp.md drops from 8,635 to 318 characters. All 169 pages that had newline structure lose all of it.

The incremental component also hardcodes max_tei_chars = 1000 and defaults chunk_size to 1200, while the full pipeline uses 600. TEI runs with --auto-truncate, so the excess is not rejected — it is dropped from the vector while the full chunk is still stored. Same mismatch #212 is fixing for the issues pipeline.

Nothing in the repo triggers incremental-pipeline.py automatically today, so this is latent rather than actively corrupting the collection. But pipelines/README.md recommends it for updates after the first full build, so the two must not disagree the moment someone follows that.
ㅤ

Suggested fix

Make utils.py the single definition of the cleaner and the URL builder, mirror both inline in the two components (a KFP component cannot import utils at runtime), and add tests that pin the copies to the shared versions.

Related: #81 (deduplicate the text-cleaning regex into a shared utility), #147 (preserve newlines in markdown cleaning).

P.S.: I'll shortly open a PR addressing this. Thanks!