argumentcomputer / argumentcomputer/Megaparsec.lean

Enhancement: Infinite streams

Open
#20 1 comment 0 reactions 0 assignees View on GitHub
difficult postponed
Dominant language
Lean
Stars
37
Forks
8
PR merge metrics
No merged PRs in 30d

Description

# Why?

In lazy languages, it's trivial to build infinite Streams. Lean would require plumbing around it to allow for infinite data structures. It's not acceptable, because we would like to parse stuff we get down the wire, signals coming from a signal generator, multipart form data.

The most practical use case is processing streams of data that are exceeding memory capabilities, such as huge JSON or XML files.

# What?

- [ ] Generate a large (1GiB+) CSV file
- [ ] Spawn a VM with 512MiB RAM
- [ ] Refactor Stream to be accommodating for a demo showing how to parse some bits out of that CSV without loading all of it into RAM
- [ ] Test it on an actual computer
- [ ] Demonstrate that it works in a small VM

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points. Start by locating the Stream implementation and the existing parsing examples, then review how a large CSV could be processed without loading it into memory. Done means demonstrating the approach in a 512MiB VM and testing it on an actual computer.

Written by the indexing model from the issue text.

Assessment

Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.