Nested dict data get silently ignored in POST request

I just spent the last several hours trying to figure out why I was able to successfully send an API POST request through curl but not through requests.

After much searching on SO, I finally figured out the solution, but I'm still confused why this should be a problem at all. Furthermore, it's very strange that requests does such a good job of simplifying HTTP requests but then reveals intricacies that an end user likely won't know how to deal with.

It turns out that requests.post() already has a json argument which handles non-trivial data dictionaries, but I don't understand why a user should need to know when to use json instead of data. To a simple user like me (and others: #2885), data simply takes a dictionary and parses it into a request. Using json in some cases but not others adds confusion and is not obvious at all. I also don't understand why I wouldn't want to always use json instead of data.

At the very least, if there's a strong reason why it makes sense to have both data and json arguments, a warning should be raised in these cases telling the user that they might not be sending what they think they're sending.

Expected Result

Here's an example request of what I was sending through curl:

$ curl -X POST 'https://httpbin.org/post' -H 'Content-Type: application/json' -d '{"first": [{"second": {"third": "data"}}]}'

{
  "args": {}, 
  "data": "{\"first\": [{\"second\": {\"third\": \"data\"}}]}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "42", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.54.0"
  }, 
  "json": {
    "first": [
      {
        "second": {
          "third": "data"
        }
      }
    ]
  }, 
  "origin": "123.56.7.890, 123.56.7.890", 
  "url": "https://httpbin.org/post"
}

Sending (what I thought was) an equivalent request through requests always returned an invalid json error:

import requests

headers = {"Content-Type": "application/json"}

data = {
    "first": [
        {
            "second": {
                "third": "data"
            }
        }
    ]
}

r = requests.post("https://httpbin.org/post", headers=headers, data=data)

print(r.json())

I expected the entire data dictionary to be sent in the request. However, when looking at the response from httpbin, it becomes clear that all the data from the second level is missing.

Actual Result

This is the response from httpbin, with almost all of the sent data missing:

{'args': {},
 'data': 'first=second',
 'files': {},
 'form': {},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Content-Length': '12',
             'Content-Type': 'application/json',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.19.1'},
 'json': None,
 'origin': '123.56.7.890, 123.56.7.890',
 'url': 'https://httpbin.org/post'}

Reproduction Steps

import requests

headers = {"Content-Type": "application/json"}

data = {
    "first": [
        {
            "second": {
                "third": "data"
            }
        }
    ]
}

r = requests.post("https://httpbin.org/post", headers=headers, data=data)

print(r.json())

Changing the request fixes the issue, but the solution is (to me) non-obvious:

r = requests.post("https://httpbin.org/post", headers=headers, json=data)

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": ""
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.6"
  },
  "platform": {
    "release": "18.5.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.19.1"
  },
  "system_ssl": {
    "version": "1000212f"
  },
  "urllib3": {
    "version": "1.23"
  },
  "using_pyopenssl": false
}

Start by running the supplied requests.post reproduction against httpbin and compare the behavior of data=data with json=data. Review the request-preparation path for how nested dictionaries are encoded and check the discussion in issue #5058. Done should mean an agreed, tested behavior for warning users or clarifying the data/json distinction.