WS response body charset does not conform to RFC 7321

Moved from https://github.com/playframework/playframework/issues/5512

Play Version (2.5.x / etc)

2.4.x - 2.5.x (fix in Future)

API (Scala / Java / Neither / Both)

Both

Expected Behavior

WS response should use the content encoding's charset where defined.

Actual Behavior

Related to https://github.com/playframework/playframework/issues/4997

So there are two methods on the response body which are still using RFC 2616 when they should be using RFC 7321 / RFC 6657:

  /**
   * The response body as String.
   */
  lazy val body: String = {
    // RFC-2616#3.7.1 states that any text/* mime type should default to ISO-8859-1 charset if not
    // explicitly set, while Plays default encoding is UTF-8.  So, use UTF-8 if charset is not explicitly
    // set and content type is not text/*, otherwise default to ISO-8859-1
    val contentType = Option(ahcResponse.getContentType).getOrElse("application/octet-stream")
    val charset = Option(HttpUtils.parseCharset(contentType)).getOrElse {
      if (contentType.startsWith("text/"))
        HttpUtils.DEFAULT_CHARSET
      else
        StandardCharsets.UTF_8
    }
    ahcResponse.getResponseBody(charset)
  }

https://github.com/playframework/playframework/blob/master/framework/src/play-ws/src/main/scala/play/api/libs/ws/ahc/AhcWS.scala#L506

and

    public String getBody() {
        try {
            // RFC-2616#3.7.1 states that any text/* mime type should default to ISO-8859-1 charset if not
            // explicitly set, while Plays default encoding is UTF-8.  So, use UTF-8 if charset is not explicitly
            // set and content type is not text/*, otherwise default to ISO-8859-1
            String contentType = ahcResponse.getContentType();
            if (contentType == null) {
                // As defined by RFC-2616#7.2.1
                contentType = "application/octet-stream";
            }
            Charset charset = HttpUtils.parseCharset(contentType);

            if (charset != null) {
                return ahcResponse.getResponseBody(charset);
            } else if (contentType.startsWith("text/")) {
                return ahcResponse.getResponseBody(HttpUtils.DEFAULT_CHARSET);
            } else {
                return ahcResponse.getResponseBody(StandardCharsets.UTF_8);
            }
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

https://github.com/playframework/playframework/blob/master/framework/src/play-java-ws/src/main/java/play/libs/ws/ahc/AhcWSResponse.java#L104

Thought about upgrading this at the same times as AHC 2.0, but it should really be a different PR.

Start with framework/src/play-ws/src/main/scala/play/api/libs/ws/ahc/AhcWS.scala and framework/src/play-java-ws/src/main/java/play/libs/ws/ahc/AhcWSResponse.java, at the response body methods identified in the issue. Compare their charset selection with RFC 7321 and RFC 6657, then check existing WS response tests. Done when both Scala and Java APIs use the defined content charset and conforming defaults.