ashvardanian / ashvardanian/StringZilla

Standard-compliant `split` implementation

Open
#98 0 comments 1 reaction 0 assignees View on GitHub
good first issue python
Dominant language
C
Stars
3.6k
Forks
135
PR merge metrics
No merged PRs in 30d

Description

The 4c738ea446cb8f9041077ac6364557527e8fc427 commit introduces a prototype for StringZilla-based Command-Line toolkit, including the `split` utility replacement. The original prototype suggests a **4x performance improvement** opportunity, but it can't currently handle multiple inputs and flags. Those must be easy to add in `cli/split.py` purely in the Python layer. We are aiming to match the following specification:

```bash
$ split --help
Usage: split [OPTION]... [FILE [PREFIX]]
Output pieces of FILE to PREFIXaa, PREFIXab, ...;
default size is 1000 lines, and default PREFIX is 'x'.

With no FILE, or when FILE is -, read standard input.

Mandatory arguments to long options are mandatory for short options too.
-a, --suffix-length=N generate suffixes of length N (default 2)
--additional-suffix=SUFFIX append an additional SUFFIX to file names
-b, --bytes=SIZE put SIZE bytes per output file
-C, --line-bytes=SIZE put at most SIZE bytes of records per output file
-d use numeric suffixes starting at 0, not alphabetic
--numeric-suffixes[=FROM] same as -d, but allow setting the start value
-x use hex suffixes starting at 0, not alphabetic
--hex-suffixes[=FROM] same as -x, but allow setting the start value
-e, --elide-empty-files do not generate empty output files with '-n'
--filter=COMMAND write to shell COMMAND; file name is $FILE
-l, --lines=NUMBER put NUMBER lines/records per output file
-n, --number=CHUNKS generate CHUNKS output files; see explanation below
-t, --separator=SEP use SEP instead of newline as the record separator;
'\0' (zero) specifies the NUL character
-u, --unbuffered immediately copy input to output with '-n r/...'
--verbose print a diagnostic just before each
output file is opened
--help display this help and exit
--version output version information and exit

The SIZE argument is an integer and optional unit (example: 10K is 10*1024).
Units are K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (powers of 1000).
Binary prefixes can be used, too: KiB=K, MiB=M, and so on.

CHUNKS may be:
N split into N files based on size of input
K/N output Kth of N to stdout
l/N split into N files without splitting lines/records
l/K/N output Kth of N to stdout without splitting lines/records
r/N like 'l' but use round robin distribution
r/K/N likewise but only output Kth of N to stdout

GNU coreutils online help:
Report any translation bugs to
Full documentation
or available locally via: info '(coreutils) split invocation'
```

Contributor guide

Open the contributing guide

Research direction

Start with the prototype introduced in commit 4c738ea446cb8f9041077ac6364557527e8fc427 and inspect cli/split.py. Compare its current command-line handling with the supplied GNU split specification, focusing on multiple inputs and flags that must be added in the Python layer. Done means the replacement accepts the documented inputs and options and matches the stated split behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.