palantir / palantir/java-compute-module

ComputeModuleClient.postResult silently drops results on transient failures

Open
#220 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
4
Forks
1
Avg merge
10h 19m
Merged PRs (30d)
17

Description

ComputeModuleClient.postResult does one client.send, catches Exception, logs, and returns. If anything goes wrong in that one call, the result is gone; the job ran fine but the user never gets it back.

Saw this on com.palantir.computemodules:lib:0.6.0 in a deployed-app Compute Module. Function completed, logged its result, and then ~30s later:

ERROR ComputeModuleClient: Failed to post result {"jobId":"..."}
java.io.IOException: HTTP/1.1 header parser received no bytes
  at jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:970)
  at com.palantir.computemodules.client.ComputeModuleClient.postResult(ComputeModuleClient.java:90)
  at com.palantir.computemodules.ComputeModule$1.onSuccess(ComputeModule.java:88)
Caused by: java.net.SocketException: Connection reset

This looks like a stale pooled keepalive: the server closed an idle connection, the JDK client grabbed it off the pool anyway, write failed before any response bytes came back.

postSchemas in the same file does retry (5 attempts, exponential backoff), but only on ConnectException. Connection reset comes through as SocketException, which is an IOException but not a ConnectException, so that loop wouldn't have caught this either.

A couple of things that would help:

  • Retry postResult on IOException. One thing to watch out for: the body is an InputStream and the supplier passed to BodyPublishers.ofInputStream is () -> result, which returns the same already-consumed stream on a retry. postSchemas sidesteps this by buffering to byte[] and using ofByteArray. Probably also worth pulling the retry loop into a shared helper.
  • The shared HttpClient is built with no tuning. The default idle-pool timeout (jdk.httpclient.keepalive.timeout) is 30s for both HTTP/1.1 and HTTP/2, so the stale-pool race is open either way. Sending Connection: close on postResult would avoid pooling for this path; bumping the keepalive system property is the other lever.

Also: postResult throws away the response status code, so an HTTP 500 from the sidecar would look exactly like a Connection reset in the logs. Worth distinguishing.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in ComputeModuleClient.java at postResult and compare its single send with the retry logic in postSchemas. Trace how the InputStream reaches BodyPublishers.ofInputStream and inspect how the shared HttpClient is built. Done means transient I/O failures are handled without losing the result and HTTP response status is distinguishable in the logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.