algolia / algolia/docsearch-scraper

Output CI-friendly progress messages

Open
#422 4 comments 0 reactions 0 assignees View on GitHub
bug enhancement help wanted
Dominant language
Python
Stars
322
Forks
115
PR merge metrics
No merged PRs in 30d

Description

Currently, the scraper assumes it's writing progress messages to an ANSI-compatible terminal. As a result, the progress messages look like this in a CI environment:

```
[94m> DocSearch: [0mhttps://docs.couchbase.com/server/6.0/introduction/intro.html ([93m51 records[0m)
[94m> DocSearch: [0mhttps://docs.couchbase.com/home/contribute/includes.html ([93m23 records[0m)
[94m> DocSearch: [0mhttps://docs.couchbase.com/server/6.0/n1ql/n1ql-language-reference/index.html ([93m28 records[0m)
```

Either add an option to output plain messages or automatically detect if ANSI color codes are not supported.

The easiest way to accomplish this might be to route the messages through a logger which can be configured separately. I'd also be interested in silencing the messages completely, which a logger would also help with.

Contributor guide

Open the contributing guide

Research direction

Start by locating where the scraper emits its progress messages, then reproduce the output in a CI or non-ANSI terminal environment. Done means progress output is readable without ANSI escape codes, with either plain-output or silencing behavior supported as appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.