projectdiscovery / projectdiscovery/utils
fileutil: add configurable options for line reading (trim, skip empty, comment filter, buffer size)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 142
- Forks
- 56
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 7
Description
Description
Add configurable options for line reading in the fileutil package.
Problem
Current ReadFile() and related functions have several limitations:
- No trimming support - cannot automatically trim whitespace from lines
- No skip empty lines - cannot filter out empty lines
- No comment filtering - cannot skip lines starting with
#(common in config files) - Poor error handling - scanner errors are silently dropped when using channel-based functions
- No buffer size control - cannot customize scanner buffer for large lines
Proposed Solution
Add new functions and types with options pattern:
LineOptiontype for configurationWithTrimSpace()— trim leading/trailing whitespaceWithSkipEmpty()— skip empty linesWithComment(prefix string)— skip lines with a comment prefix (e.g.,#)WithBufferSize(size int)— custom scanner buffer size
Two new functions:
ReadFileWithError()— reads a file with proper error propagation via two channels (lines + errors)ReadLinesStream()— reads a file with all configurable options
Existing functions will remain unchanged for backward compatibility.
Benefits
- Better error handling (no more silent failures)
- More flexible file processing
- Backward compatible with existing code
- Useful for all ProjectDiscovery tools (Subfinder, HTTPX, Nuclei, Katana, etc.)
- Follows Go best practices with options pattern
Example Usage
linesCh, errCh := fileutil.ReadLinesStream("file.txt",
fileutil.WithTrimSpace(),
fileutil.WithSkipEmpty(),
fileutil.WithComment("#"),
fileutil.WithBufferSize(1024*1024),
)
for {
select {
case line, ok := <-linesCh:
if !ok {
select {
case err, ok := <-errCh:
if ok && err != nil {
return err
}
default:
}
return nil
}
// process line
case err, ok := <-errCh:
if !ok {
return nil
}
return err
}
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the fileutil package's existing ReadFile and related channel-based functions, then review how they handle scanner errors and line processing. Define the option behavior and add ReadFileWithError and ReadLinesStream so trimming, empty-line skipping, comment filtering, buffer sizing, and error propagation work as described while existing functions remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100