iter_lines method will always hold the last response in the server in a buffer

The implementation of the iter_lines and iter_content methods in requests means that when receiving line-by-line data from a server in "push" mode, the latest line received from the server will almost invariably be smaller than the chunk_size parameter, causing the final read operation to block.

A good example of this is the Kubernetes watch api, which produces one line of JSON output per event, like this:

{"type":"ADDED","object":{"kind":"Service","id":"kubernetes-ro","uid":"5718d954-a31a-11e4-8c74-20cf30467e62","creationTimestamp":"2015-01-23T11:10:25-05:00","selfLink":"/api/v1beta1/services/kubernetes-ro","resourceVersion":4,"apiVersion":"v1beta1","namespace":"default","port":80,"protocol":"TCP","labels":{"component":"apiserver","provider":"kubernetes"},"selector":null,"containerPort":0,"portalIP":"10.254.6.100"}}

If you compare the output of:

import requests
r = requests.get('http://localhost:8080/api/v1beta1/watch/services',
                 stream=True)

for line in r.iter_lines():
    print line

With the output of curl running against the same URL, you will see
that the output from the Python code lags behind the output seen by
curl by one line.

I was able to work around this behavior by writing my own iter_lines
method, which looks like this:

def iter_lines(fd, chunk_size=1024):
    '''Iterates over the content of a file-like object line-by-line.'''

    pending = None

    while True:
        chunk = os.read(fd.fileno(), chunk_size)
        if not chunk:
            break

        if pending is not None:
            chunk = pending + chunk
            pending = None

        lines = chunk.splitlines()

        if lines and lines[-1]:
            pending = lines.pop()

        for line in lines:
            yield line

    if pending:
        yield(pending)

This works around the problem partly by calling os.read, which will
happily return fewer bytes than requested in chunk_size. With the
above routing available, the following code behaves correctly:

import requests
r = requests.get('http://localhost:8080/api/v1beta1/watch/services',
                 stream=True)

for line in iter_lines(r.raw):
    print line

This code will always print out the most recent reply from the server
when it is received.

Start by reading the implementations of requests' iter_lines and iter_content, then reproduce the streaming Kubernetes watch example and compare its output with curl. Done means the iterator yields each received line without lag while still handling chunk boundaries and the final buffered line correctly.