tornadoweb / tornadoweb/tornado

Highlight stream_request_body in documentation

Open
#2,952 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

docs web
Dominant language
Python
Stars
22.2k
Forks
5.6k
Avg merge
3h 42m
Merged PRs (30d)
16

Description

I am pretty sure this has been addressed somewhere else but I can't find a solution, so I am asking for help here.

I have a POST to upload files into my server as follows:

class uploadHandler(BaseHandler):
    def post(self):
        if(checkForwardedHeader(self.request.headers)):
            posts.upload(self.request.files['file'][0])
        else:
            raise tornado.web.HTTPError(403) # Unauthorized!

(Base handler is a custom RequestHandler that enables POSTS, GET and OPTIONS, nothing else, so no relevant here)

def upload(filenames):
    try:
      path=xxxxxxxxx #customapathusingthe filenames['filename'] value
      filebody=filenames['body']
      
      output_file = open(path, 'wb')
      output_file.write(filebody)

      # close file
      output_file.close()
    exception Exception as e:
        .............. # capturing possible errors

This works completely fine with small files (<100Mb), 0 problems

However, for larger files, it does not work, even though Nginx should set the max_body_size to 800M

Uploads return, for large files: [I 201110 12:37:48 http1connection:289] Malformed HTTP message from 127.0.0.1: Content-Length too long
With Nginx capturing the following:

[error] 838512#838512: *3332387 readv() failed (104: Connection reset by peer) while reading upstream, client: 127.0.0.1, server: , request: "POST /upload HTTP/1.0", upstream: "http://127.0.0.1:8888/upload", host: "xxxxxx.com", referrer: "https://xxxxx.com"

So I decided to run the application with a custom max_buffer_size bigger than 100Mb to see if that was the issue since the docs (https://www.tornadoweb.org/en/stable/httpclient.html?highlight=upload) says the default size for the buffer is 100Mb

http_server = tornado.httpserver.HTTPServer(application, max_buffer_size=800000000) # 800MB buffer
http_server.listen(options.port)
tornado.ioloop.IOLoop.current().start()

Upon doing this, the error message changed to 502 Bad Gateway and immediatly crashing the webserver with a "Killed" message when uploading a file.
Ngninx logs show, in the other hand the following captured error:

2020/11/10 12:34:41 [error] 838512#838512: *3309856 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: , request: "POST /upload HTTP/1.0", upstream: "http://127.0.0.1:8888/upload", host: "xxxxx.com", referrer: "https://xxxxx.com"

Everything works fine for uploads < 100MB, fails for even 101MB :)

How should I actually approach big files uploads with Tornado?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the HTTP client documentation linked in the issue and the Tornado documentation for stream_request_body. Clarify how large uploads are handled and what setting or approach the example should use; done means a newcomer can understand the documented limit and recommended upload path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.