gchq / gchq/sleeper

Receive queried rows through CloseableIterator as they come from web socket

Open
#6,463 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a developer working with Sleeper, I want rows received from a web socket to be returned to my code immediately, so that I can start processing before all the rows are returned.

### Description / Background

Follows on from:
- https://github.com/gchq/sleeper/issues/5879
- https://github.com/gchq/sleeper/issues/6499

The linked issue created a CloseableIterator suitable for wiring into QueryWebSocketListener. The listener still waits for all the sub-queries to finish before returning any rows to the user, and gathers them into one big list.

We'd like to return rows through that CloseableIterator as they come from the web socket.

### Acceptance Criteria

**Given** a query made to the SleeperClient or QueryWebSocketClient via a web socket
**When** there are multiple batches of rows returned in separate, spaced out web socket messages
**Then** an iterator returns the rows as soon as a batch is returned, so they aren't all held in memory at once

### Technical Notes / Implementation Details

QueryWebSocketListener currently caches the received rows in a `Map> queryIdToRows`. If we pass them directly to the CloseableIterator, that won't be necessary.

Given that the server can send us rows at any time, there's no guarantee that we'll be able to process all the rows. Our implementation of the iterator will be something like a wrapper around a buffer, which can hold all the rows, but can delete them as they are consumed.

The messages will come in through a WebSocketClient, as implemented in QueryWebSocketConnection. This runs a single thread that listens for messages from the socket, which is started when we call WebSocketClient.connect through QueryWebSocketConnection.initialiseConnection. This means that we only need to worry about two threads - the one receiving the web socket messages, and the one retrieving the rows.

We'll need some sort of queue to sit between the thread receiving the web socket messages and the thread receiving the rows.

#### Queue implementation

Inside the CloseableIterator, we could use a Java BlockingQueue to hold some parsed version of the web socket messages. We could leave QueryWebSocketListener tracking which queries are in progress as it does now, and pass a simplified version of the messages to the CloseableIterator. The messages can be either a batch of rows, an exception, or a notification that it's finished. We can keep all of these on a BlockingQueue and then resolve them into local state for the iterator in the thread receiving the rows, when it calls hasNext or next.

#### Iterator state

Ideally we could avoid ever copying the lists of rows as they come from the web socket. When we take a message off the queue that contains a batch of rows, we can take the list directly from that batch, the same object that came from the web socket, and have another field that tracks where we're up to in that list. That way we can return one row at a time directly from that list.

### Dependencies / Blockers

Depends on:
- https://github.com/gchq/sleeper/issues/5879

Contributor guide

Open the contributing guide

Research direction

Start with QueryWebSocketListener and the CloseableIterator created by issue 5879, then trace message handling through QueryWebSocketConnection.initialiseConnection and WebSocketClient.connect. Review how queryIdToRows is populated and how the two socket and consumer threads interact. Done means spaced batches are available through the iterator as they arrive without retaining all rows in one list, while exceptions and completion are handled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.