spring-projects / spring-projects/spring-framework
Streaming parsing of JSON array in Spring WebClient
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 60.2k
- Forks
- 38.8k
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 27
Description
I found #21862 which is pretty close to my request but closed.
I am currently using Spring WebClient with Spring Boot 2.2.6 and Spring Framework 5.2.5 writing a service that sits in front of a number of other upstream services and transforms their response for public consumption. Some of these services respond with very large JSON payloads that are little more than an array of entities wrapped in a JSON document, usually with no other properties:
{
"responseRoot": {
"entities": [
{ "id": "1" },
{ "id": "2" },
{ "id": "n" },
]
}
}
There could be many thousands of entities in this nested array and the entire payload can be tens of MBs. I want to be able to read in these entities through a Flux<T> so that I can transform them individually and write them out to the client without having to deserialize all of them into memory. This doesn't appear to be something that Spring WebFlux supports out of the box.
I'm currently exploring writing my own BodyExtractor which reuses some of the code in Jackson2Tokenizer to try to support this. My plan is to accept a JsonPointer to the location of the array and then parse asynchronously until I find that array, then to buffer the tokens for each array element to deserialize them.
var flux = client.get()
.uri(uri)
.exchange()
.flatMapMany(r ->
r.body(new StreamingBodyExtractor(JsonPointer.compile("/responseRoot/entities")))
);
Before I go too far down this path I was curious if this was functionality that Spring would be interested in supporting out of the box.
Similarly, I was curious about the functionality of being able to stream out a response from a WebFlux controller via a Flux<T> where the streamed response would be wrapped in a JSON array and possibly in a root JSON document as well?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the WebClient body extraction flow and the existing Jackson2Tokenizer code mentioned in the issue. Investigate how a JsonPointer could identify a nested array and how a WebFlux controller currently handles Flux responses. Done would mean a clear supported approach for streaming array elements in and emitting them in a wrapped JSON response.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring
- Domain
- api, backend, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100