Is it possible to avoid reallocation during chunked data processing?
- Dominant language
- C
- Stars
- 2.1k
- Forks
- 274
- PR merge metrics
- No merged PRs in 30d
Description
For example, if i'm receiving huge 200mb file in "multipart/form-data" and i don't want to expand buffer to such enormous size. What I could do?
I was looking at https://github.com/h2o/picohttpparser/issues/14 but it seems to be not an appropriate solution in my case. In comparison to example in README.md you just removed part with realloc() but will function process data properly in that case? I don't get it.
The only one solution that comes to my head it to:
1) read() and immediately write() all incoming data to temprorary file in disk
2) mmap() whole file to memory
3) pass whole mmap()ped region to phr_decode_chunked() and decode everything with one pass
4) write result(s) to files
5) remove temp file
That's kinda stable and memory-tolerant solution, but i wish there was a way to avoid redundant disk read()/write() and avoid execution of background tasks.
P.S. I'd like ask one more question. I want to use your library with FastCGI, where http headers are already parsed by web server and available via FastCGI api. Is it possible to use phr_parse_response() without previous using phr_parse_headers()?
Before i found your library i was almost stared own small library for parsing POST and POST in multipart form data, which is quite hard if you are using small and fixed size buffer. But I really don't want to do that :D
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with README.md and the documented behavior of phr_decode_chunked(), phr_parse_response(), and phr_parse_headers(). Clarify whether the requested work is a streaming chunked-decoding API, FastCGI header support, or guidance for existing APIs; completion would require a defined scope and expected behavior from the maintainer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100