NullPointerException in ring.middleware.multipart-params middleware

Hi there,

We are getting an NPE thrown from the ring.middleware.multipart-params and I don't know where to start looking to track the problem down. Any help would be appreciated.

This happens very rarely, 19 times in the last 7 days, on a HTTP endpoint that is accessed many, many thousands of times per day.

The exception is thrown from the org.apache.commons.fileupload library, which is called from ring.middleware.multipart-params. There are a couple of our middleware items on the callstack, but I don't think these are very likely to be the cause.

Callstack

class java.lang.NullPointerException
File "MultipartStream.java", line 999 in org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable
File "MultipartStream.java", line 943 in org.apache.commons.fileupload.MultipartStream$ItemInputStream.close
File "MultipartStream.java", line 922 in org.apache.commons.fileupload.MultipartStream$ItemInputStream.close
File "IOUtils.java", line 339 in org.apache.commons.io.IOUtils.closeQuietly
File "IOUtils.java", line 270 in org.apache.commons.io.IOUtils.closeQuietly
File "Streams.java", line 123 in org.apache.commons.fileupload.util.Streams.copy
File "Streams.java", line 70 in org.apache.commons.fileupload.util.Streams.copy
File "MultipartStream.java", line 593 in org.apache.commons.fileupload.MultipartStream.readBodyData
File "MultipartStream.java", line 617 in org.apache.commons.fileupload.MultipartStream.discardBodyData
File "MultipartStream.java", line 634 in org.apache.commons.fileupload.MultipartStream.skipPreamble
File "FileUploadBase.java", line 1023 in org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.findNextItem
File "FileUploadBase.java", line 1003 in org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.<init>
File "FileUploadBase.java", line 310 in org.apache.commons.fileupload.FileUploadBase.getItemIterator
File "multipart_params.clj", line 46 in ring.middleware.multipart-params/file-item-seq
File "multipart_params.clj", line 42 in ring.middleware.multipart-params/file-item-seq
File "multipart_params.clj", line 63 in ring.middleware.multipart-params/parse-multipart-params
File "multipart_params.clj", line 59 in ring.middleware.multipart-params/parse-multipart-params
File "multipart_params.clj", line 91 in ring.middleware.multipart-params/multipart-params-request
File "multipart_params.clj", line 80 in ring.middleware.multipart-params/multipart-params-request
File "RestFn.java" line 423 in clojure.lang.RestFn.invoke
File "multipart_params.clj", line 118 in ring.middleware.multipart-params/wrap-multipart-params[fn]
File "ssl.clj", line 37 in circle.http.ssl/wrap-force-ssl[fn]
File "methods.clj", line 10 in circle.http.methods/wrap-deny-unacceptable-methods[fn]
File "exceptions.clj", line 107 in circle.http.exceptions/wrap-exceptions[fn]
File "errors.clj", line 62 in circle.web.http.errors/wrap-status-pages[fn]
File "logging.clj", line 209 in circle.http.logging/wrap-logging[fn]
File "logging.clj", line 208 in circle.http.logging/wrap-logging[fn]
File "core.clj", line 31 in circleci.request-trace.core/wrap-request-id[fn]
File "RingHandler.java", line 91 in org.httpkit.server.HttpHandler.run
File "Executors.java" line 511 in java.util.concurrent.Executors$RunnableAdapter.call
File "FutureTask.java" line 266 in java.util.concurrent.FutureTask.run
File "ThreadPoolExecutor.java" line 1149 in java.util.concurrent.ThreadPoolExecutor.runWorker
File "ThreadPoolExecutor.java" line 624 in java.util.concurrent.ThreadPoolExecutor$Worker.run
File "Thread.java" line 748 in java.lang.Thread.run

Some context we have gathered:

headers.accept: */*
headers.host: circleci.com
headers.user-agent: PostmanRuntime/7.4.0
(our internal) request_id: 410e1f74-63a8-4f96-b145-df7147fca973
request_method: get
url: http://circleci.com:80/api/v1.1/recent-builds
timestamp: 1544718529 - 2018-12-13 16:28:49

In all cases, the User Agent is either PostmanRuntime/7.4.0 or PostmanRuntime/7.3.0.

Start at ring.middleware.multipart-params/file-item-seq and parse-multipart-params in multipart_params.clj, then inspect the Apache Commons FileUpload MultipartStream stack shown. Use the supplied request context and Postman user-agent details to investigate a reproducible failure; done means identifying a cause and recording a verified fix or clear diagnosis.