kubeflow / kubeflow/docs-agent
refactor(pipelines): Deduplicate text-cleaning regex into shared utility
- Dominant language
- Python
- Stars
- 42
- Forks
- 111
- Avg merge
- 6d 23m
- Merged PRs (30d)
- 2
Description
## Problem
Both `pipelines/kubeflow-pipeline.py` (L246-L268) and `pipelines/incremental-pipeline.py` (L171-L193) share a byte-for-byte identical block of 8 regex substitutions for cleaning Hugo frontmatter, template syntax, HTML tags, and markdown artifacts.
Currently, if the cleaning logic needs to be updated, it must be updated in multiple places, violating DRY and risking divergence.
- Issue #49 only covers extracting shared config for the API servers, not the ingestion pipelines.
## Proposed Solution
Extract this logic into a `clean_content(text: str) -> str` function in a new `shared/text_utils.py` module, and update both pipeline components to import and use it.
## Related
- GSoC Pre-work: Part of codebase cleanup.
- Tracker: #72
Contributor guide
Research direction
Read the duplicated regex blocks in pipelines/kubeflow-pipeline.py (L246-L268) and pipelines/incremental-pipeline.py (L171-L193), then inspect the surrounding imports and the proposed shared/text_utils.py location. Done means both pipelines use clean_content(text: str) -> str from the shared module while preserving the existing cleaning behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100