projectdiscovery / projectdiscovery/utils

fileutil: add configurable options for line reading (trim, skip empty, comment filter, buffer size)

Open
#755 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
142
Forks
56
Avg merge
2d 10h
Merged PRs (30d)
7

Description

Description

Add configurable options for line reading in the fileutil package.

Problem

Current ReadFile() and related functions have several limitations:

  1. No trimming support - cannot automatically trim whitespace from lines
  2. No skip empty lines - cannot filter out empty lines
  3. No comment filtering - cannot skip lines starting with # (common in config files)
  4. Poor error handling - scanner errors are silently dropped when using channel-based functions
  5. No buffer size control - cannot customize scanner buffer for large lines

Proposed Solution

Add new functions and types with options pattern:

  • LineOption type for configuration
  • WithTrimSpace() — trim leading/trailing whitespace
  • WithSkipEmpty() — skip empty lines
  • WithComment(prefix string) — skip lines with a comment prefix (e.g., #)
  • WithBufferSize(size int) — custom scanner buffer size

Two new functions:

  1. ReadFileWithError() — reads a file with proper error propagation via two channels (lines + errors)
  2. ReadLinesStream() — reads a file with all configurable options

Existing functions will remain unchanged for backward compatibility.

Benefits

  • Better error handling (no more silent failures)
  • More flexible file processing
  • Backward compatible with existing code
  • Useful for all ProjectDiscovery tools (Subfinder, HTTPX, Nuclei, Katana, etc.)
  • Follows Go best practices with options pattern

Example Usage

linesCh, errCh := fileutil.ReadLinesStream("file.txt",
    fileutil.WithTrimSpace(),
    fileutil.WithSkipEmpty(),
    fileutil.WithComment("#"),
    fileutil.WithBufferSize(1024*1024),
)

for {
    select {
    case line, ok := <-linesCh:
        if !ok {
            select {
            case err, ok := <-errCh:
                if ok && err != nil {
                    return err
                }
            default:
            }
            return nil
        }
        // process line
    case err, ok := <-errCh:
        if !ok {
            return nil
        }
        return err
    }
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the fileutil package's existing ReadFile and related channel-based functions, then review how they handle scanner errors and line processing. Define the option behavior and add ReadFileWithError and ReadLinesStream so trimming, empty-line skipping, comment filtering, buffer sizing, and error propagation work as described while existing functions remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.