tornadoweb / tornadoweb/tornado
Highlight stream_request_body in documentation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 22.2k
- Forks
- 5.6k
- Avg merge
- 3h 42m
- Merged PRs (30d)
- 16
Description
I am pretty sure this has been addressed somewhere else but I can't find a solution, so I am asking for help here.
I have a POST to upload files into my server as follows:
class uploadHandler(BaseHandler):
def post(self):
if(checkForwardedHeader(self.request.headers)):
posts.upload(self.request.files['file'][0])
else:
raise tornado.web.HTTPError(403) # Unauthorized!
(Base handler is a custom RequestHandler that enables POSTS, GET and OPTIONS, nothing else, so no relevant here)
def upload(filenames):
try:
path=xxxxxxxxx #customapathusingthe filenames['filename'] value
filebody=filenames['body']
output_file = open(path, 'wb')
output_file.write(filebody)
# close file
output_file.close()
exception Exception as e:
.............. # capturing possible errors
This works completely fine with small files (<100Mb), 0 problems
However, for larger files, it does not work, even though Nginx should set the max_body_size to 800M
Uploads return, for large files: [I 201110 12:37:48 http1connection:289] Malformed HTTP message from 127.0.0.1: Content-Length too long
With Nginx capturing the following:
[error] 838512#838512: *3332387 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: , request: "POST /upload HTTP/1.0", upstream: "http://127.0.0.1:8888/upload", host: "xxxxxx.com", referrer: "https://xxxxx.com"
So I decided to run the application with a custom max_buffer_size bigger than 100Mb to see if that was the issue since the docs (https://www.tornadoweb.org/en/stable/httpclient.html?highlight=upload) says the default size for the buffer is 100Mb
http_server = tornado.httpserver.HTTPServer(application, max_buffer_size=800000000) # 800MB buffer
http_server.listen(options.port)
tornado.ioloop.IOLoop.current().start()
Upon doing this, the error message changed to 502 Bad Gateway and immediatly crashing the webserver with a "Killed" message when uploading a file.
Ngninx logs show, in the other hand the following captured error:
2020/11/10 12:34:41 [error] 838512#838512: *3309856 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: , request: "POST /upload HTTP/1.0", upstream: "http://127.0.0.1:8888/upload", host: "xxxxx.com", referrer: "https://xxxxx.com"
Everything works fine for uploads < 100MB, fails for even 101MB :)
How should I actually approach big files uploads with Tornado?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the HTTP client documentation linked in the issue and the Tornado documentation for stream_request_body. Clarify how large uploads are handled and what setting or approach the example should use; done means a newcomer can understand the documented limit and recommended upload path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100