tornadoweb / tornadoweb/tornado

CPU 100% with tornado 4.5.3?

Open
#2,373 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

iostream
Dominant language
Python
Stars
22.2k
Forks
5.6k
Avg merge
3h 42m
Merged PRs (30d)
16

Description

We have built a app with tornado 4.5.3, along with pynsq 0.8.2(which was not compatible with tornado 5), running on CentOS 7.4 with default kernel 3.10.0-693.21.1.el7.x86_64.
CPU usage came up to 100% soon after app started, with ~100 connections.
We are using tornado's TCPServer class and IOStream.read_bytes(4096, partial=True) to implement our async tcp server, something like

        while True:
            try:
                buf = await self._stream.read_bytes(4096, partial=True)
            except:
                self.close()
                return None
            return buf

Many lines like read(370, 0x2e47d80, 11) = -1 EAGAIN (Resource temporarily unavailable) in strace result, and strace -c result was:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 37.95    0.892246           4    245309    243731 read
 35.95    0.845239           3    242459         2 ioctl
 19.15    0.450160           4    119424           epoll_wait
  4.58    0.107586          13      7996       590 futex
  0.46    0.010713           3      3263      1395 recvfrom
  0.43    0.010007           5      1908           sendto
  0.33    0.007788           3      2700           epoll_ctl
  0.30    0.006951           4      1638           write
  0.19    0.004412           3      1746           fstat
  0.17    0.004001           2      1711      1705 lseek
  0.10    0.002335           3       754       726 getpeername

A tornado issue?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the TCPServer usage and the IOStream.read_bytes(4096, partial=True) call described in the report, then reproduce the behavior with the stated Tornado, pynsq, CentOS, and kernel versions. Compare the event-loop activity with the repeated EAGAIN reads and epoll_wait calls; done means determining whether Tornado is responsible and documenting a reproducible cause or required configuration change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.