kubeflow / kubeflow/docs-agent

refactor(pipelines): Deduplicate text-cleaning regex into shared utility

Open
#81 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
42
Forks
111
Avg merge
6d 23m
Merged PRs (30d)
2

Description

## Problem
Both `pipelines/kubeflow-pipeline.py` (L246-L268) and `pipelines/incremental-pipeline.py` (L171-L193) share a byte-for-byte identical block of 8 regex substitutions for cleaning Hugo frontmatter, template syntax, HTML tags, and markdown artifacts.

Currently, if the cleaning logic needs to be updated, it must be updated in multiple places, violating DRY and risking divergence.
- Issue #49 only covers extracting shared config for the API servers, not the ingestion pipelines.

## Proposed Solution
Extract this logic into a `clean_content(text: str) -> str` function in a new `shared/text_utils.py` module, and update both pipeline components to import and use it.

## Related
- GSoC Pre-work: Part of codebase cleanup.
- Tracker: #72

Contributor guide

Open the contributing guide

Research direction

Read the duplicated regex blocks in pipelines/kubeflow-pipeline.py (L246-L268) and pipelines/incremental-pipeline.py (L171-L193), then inspect the surrounding imports and the proposed shared/text_utils.py location. Done means both pipelines use clean_content(text: str) -> str from the shared module while preserving the existing cleaning behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.