Add on_line callback
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 4.3k
- Forks
- 299
- Avg merge
- 16h 19m
- Merged PRs (30d)
- 1
Description
Summary
Add an on_line event which is called for every line. This would allow for custom processing of lines even if they are skipped, comments, or used as a column headings.
Motivation
I have a file which starts with the text "Generated Date: MM/DD/YYYY" on the first line. Using the Stream API, I can skip this line with from_line:1 and everything works great. However, I would like to be able to read this line and parse out the date.
Alternative
I tried using on_record to parse the date I need and then return null for the first row. However, setting columns:true seems to try and use the first row as column names without first passing it through on_record and skipping it.
Draft
Option on_line
The on_line option provides an option to alter and filter lines. It expects a function which receives the line and a context as arguments and which returns the new altered line or nothing if the line is to be filtered.
Type: function
Optional
Default: undefined
Related: on_record, info — see Available Options
This option works at the line level. It complements the on_record option which is adapted to record-level transformations. Also, the stream-transform package provides more advanced control on the record and stream of records with asynchronous execution and concurrent control.
Use cases
Use this option to filter, enrich, and apply any transformations on a line.
Usage
The option takes a function which is called with two arguments: the input line and the context. The return value is the new line or is filtered if null or undefined are returned.
Altering lines
In the [alter line example], Member# is replaced with Member Number in the header row.
parser.on('line', function(line, {lines}){
if(lines === 1) {
return line.replace("Member#", "Member Number")
}
});
parser.write('Member#, First Name, Last Name\n');
parser.write('12345, Bill, Waterson\n');
Filtering lines
In the [filter lines example], the function returns null for the first line, filtering it from the result, but allows us to parse the Generated Date.
parser.on('line', function(line, {lines}){
if(lines === 1) {
return line.replace("Member#", "Member Number")
}
});
parser.write('Date Generated: March 5, 2000\n');
parser.write('Member#, First Name, Last Name\n');
parser.write('12345, Bill, Waterson\n');
Additional context
Has some similarities to #292, though I feel this is a more generic solution.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Stream API and the existing on_record and info option behavior described in the issue. Verify that an on_line callback can inspect every line, alter or filter it before record handling, and that the documented examples and available-options entry match the implemented behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100