twisted / twisted/treq

Inconsistent handling of data sequence values with/without files

Open
#360 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
603
Forks
140
PR merge metrics
No merged PRs in 30d

Description

import os

from twisted.internet.task import react

import treq


async def main(reactor):
    print("Without files:")
    response = await treq.post(
        "https://httpbin.org/post",
        data={"form": ["1", "2"]},
    )
    print(await response.text())

    print("With files:")
    response = await treq.post(
        "https://httpbin.org/post",
        data={"form": ["1", "2"]},
        files={"foo": (os.devnull, open(os.devnull, "rb"))},
    )
    print(await response.text())


react(main, [])

When run:

Without files:
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "form": [
      "1", 
      "2"
    ]
  }, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Content-Length": "13", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "X-Amzn-Trace-Id": "Root=1-63a26dee-580ab8e77fdd228c1eb84266"
  }, 
  "json": null, 
  "origin": "99.187.228.202", 
  "url": "https://httpbin.org/post"
}

With files:
main function encountered error
Traceback (most recent call last):
  File ".../twisted/internet/defer.py", line 696, in callback
    self._startRunCallbacks(result)
  File ".../twisted/internet/defer.py", line 798, in _startRunCallbacks
    self._runCallbacks()
  File ".../twisted/internet/defer.py", line 892, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File ".../twisted/internet/defer.py", line 1792, in gotResult
    _inlineCallbacks(r, gen, status, context)
--- <exception caught here> ---
  File ".../twisted/internet/defer.py", line 1697, in _inlineCallbacks
    result = context.run(gen.send, result)
  File ".../demo.py", line 17, in main
    response = await treq.post(
  File ".../src/treq/api.py", line 32, in post
    return _client(kwargs).post(url, data=data, _stacklevel=4, **kwargs)
  File ".../src/treq/client.py", line 182, in post
    return self.request('POST', url, data=data, **kwargs)
  File ".../src/treq/client.py", line 244, in request
    bodyProducer, contentType = self._request_body(data, files, json,
  File ".../src/treq/client.py", line 379, in _request_body
    multipart.MultiPartProducer(data + files, boundary=boundary),
  File ".../src/treq/multipart.py", line 53, in __init__
    self._fields = list(_sorted_by_type(_converted(fields)))
  File ".../src/treq/multipart.py", line 350, in _sorted_by_type
    return sorted(fields, key=key)
  File ".../src/treq/multipart.py", line 260, in _converted
    raise ValueError(
builtins.ValueError: Expected tuple: (filename, content type, producer)

When only data is passed we URL-encode the request body with urllencode(..., doseq=True), whereas when both data and files are passed we combine them and pass the result to MultiPartProducer without unboxing sequences of values.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/treq/client.py at _request_body and follow the multipart path into src/treq/multipart.py, especially the handling of converted fields. Reproduce the data-and-files example from the issue, then verify that sequence values are handled consistently in multipart requests without the reported ValueError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.