openSUSE / openSUSE/supportutils

sed performance issues

Open
#187 4 comments 0 reactions 1 assignee View on GitHub

@g23guy is already working on this.

Since Oct 28, 2024.

bug
Dominant language
Shell
Stars
35
Forks
70
Avg merge
5d 16h
Merged PRs (30d)
1

Description

Realizing that during "Updates..." the sed process consumes "100% CPU" for several minutes, I investigated it a bit (see also https://stackoverflow.com/q/77818891/6607497). Eventually I could reduce the initial runtime of more than six minutes to less than half of a second.

Note that similar sed code it used in other places, too.

Some comments from the question cited above:
(...) And it is obvious that at least 9 of the s/// can be trivially refactored into just one. –

It also appears to rewrite the same line multiple times. For example, if I change the /g to /gp on each line, it prints three (identical) lines for the input password = foo bar –

not relevant to the problem but [P|p] probably doesn't match what is intended. –

Honestly, here, in most of replacement cases, a regexp is not even needed : having multiple regexps .*word.* is fundamentally inefficient (by several order of magnitude). Indeed, one can just search a bag of words in each line and replace the whole line. The later is far more efficient. –

Technically, .*foo.* is not the same as foo.* but that seems unlikely to be a issue here. As an aside, any s/// that matches foo.* does not need /g since there can be no other matches, but removing it won't improve performance here. –

(.*foo.* matches the final occurrence of "foo"; foo.* matches the first occurrence - but if the input ever contains two occurrences (eg. wanted1 foo secret wanted2 foo secret), the original code would fail to sanitize fully (wanted1 foo secret wanted2 foo hidden, while with the change it would (wanted1 foo hidden))

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.