ring-clojure / ring-clojure/ring

Params middleware does not decode some strings correctly

Open
#269 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Clojure
Stars
3.9k
Forks
528
PR merge metrics
No merged PRs in 30d

Description

Hi James, thanks for the continued work on maintaining ring.

Just want to throw this issue out there in case anyone else stumbles upon the same issue, plus gathering some feedback.

In our project, we were dealing with non utf-8 encoded forms (Shift_JIS to be specific), and found that the params middleware is garbling the incoming string, even though we were using the wrap-params middleware with the Shift_JIS option e.g. (param/wrap-params {:encoding "Shift_JIS"})

Looking at the source code deeper inside, I see that java.net.URLDecoder is being used for parsing, and that java.net.URLDecoder does not decode some non utf-8 strings correctly, while org.apache.commons.codec.net.URLCodec can, illustrated in below snippet.

  ;; "モジバケコワイ", URL encoded with Shif_JIS by the browser
  "%83%82%83W%83o%83P%83R%83%8F%83C"

 (import [java.net URLDecoder])
 (URLDecoder/decode "%83%82%83W%83o%83P%83R%83%8F%83C" "Shift-JIS")
 ;; => "モ�W�o�P�Rワ�C" 

 (import [org.apache.commons.codec.net URLCodec])
 (let [codec (URLCodec. "Shift-JIS")]
   (.decode codec "%83%82%83W%83o%83P%83R%83%8F%83C" "Shift-JIS"))
 ;; => "モジバケコワイ"

We came up with 2 work arounds for this issue:

  1. Use org.apache.commons.codec.net.URLCodec instead of java.net.URLDecoder for decoding URL encoded parameteres
  2. Use a form with an enctype of multipart/form-data so that nothing gets encoded and thus avoid the problem entirely

Would appreciate the if I can get feed back on:

  1. Which of the above workaround is preferable?
  2. Would you be interested in a PR that replaces the decoder used in the params middleware with org.apache.commons.codec.net URLCodec?

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the params middleware's URL-decoding path and the wrap-params encoding option described in the issue. Reproduce the Shift_JIS example, compare the decoder behavior, and confirm completion when the encoded parameter yields the original Japanese string without breaking existing parameter parsing.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.