h2o / h2o/picohttpparser

Is it possible to avoid reallocation during chunked data processing?

Open
#48 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
2.1k
Forks
274
PR merge metrics
No merged PRs in 30d

Description

For example, if i'm receiving huge 200mb file in "multipart/form-data" and i don't want to expand buffer to such enormous size. What I could do?

I was looking at https://github.com/h2o/picohttpparser/issues/14 but it seems to be not an appropriate solution in my case. In comparison to example in README.md you just removed part with realloc() but will function process data properly in that case? I don't get it.

The only one solution that comes to my head it to:

1) read() and immediately write() all incoming data to temprorary file in disk
2) mmap() whole file to memory
3) pass whole mmap()ped region to phr_decode_chunked() and decode everything with one pass
4) write result(s) to files
5) remove temp file

That's kinda stable and memory-tolerant solution, but i wish there was a way to avoid redundant disk read()/write() and avoid execution of background tasks.

P.S. I'd like ask one more question. I want to use your library with FastCGI, where http headers are already parsed by web server and available via FastCGI api. Is it possible to use phr_parse_response() without previous using phr_parse_headers()?

Before i found your library i was almost stared own small library for parsing POST and POST in multipart form data, which is quite hard if you are using small and fixed size buffer. But I really don't want to do that :D

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with README.md and the documented behavior of phr_decode_chunked(), phr_parse_response(), and phr_parse_headers(). Clarify whether the requested work is a streaming chunked-decoding API, FastCGI header support, or guidance for existing APIs; completion would require a defined scope and expected behavior from the maintainer.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.