AbsaOSS / AbsaOSS/atum-service

Enhance measurement querying

Abierto
#478 1 comentario 1 reacción 1 asignado Reclamado por @lsulak Ver en GitHub
enhancement
Lenguaje dominante
Scala
Estrellas
6
Forks
1
Merge medio
2 d 7 h
PR fusionados (30 d)
4

Descripción

# Atum reader requirements for measurement querying in Unify

This document captures limitations in the Atum `atum-reader` public API that force
awkward workarounds in `DataMeasurementService`
(`contexts/data-measurement/implementation`), and the changes we would like in the reader
to support the querying we need cleanly.

## Context

We store measurement checkpoints in the Atum service. Each checkpoint has:

- a **name** (e.g. the domain final checkpoint name `"Domain feeds data written"`), and
- **properties**, including `executionID` (written by the spark plugins via
`AtumSupport.Checkpoints.defaultCheckpointProperties`).

Two service queries need to filter checkpoints:

1. `listDomainMajorVersionMeasurements` — may filter by `executionIDs` (set) OR by
`checkpointName`, plus paginate.
2. `listDomainMajorVersionFinalMeasurements` — needs the latest checkpoint of a fixed
**name** AND (ideally) restricted to given `executionIDs`.

## Current reader API (as of `atum-reader_2.12` 0.8.0)

Both `FlowReader` and `PartitioningReader` expose three mutually-exclusive public methods:

| method | filters by |
| ----------------------------- | --------------------- |
| `getCheckpointsPage` | nothing (all) |
| `getCheckpointsOfNamePage` | checkpoint **name** |
| `getCheckpointsByPropertiesPage` | checkpoint **properties** (single key→value map) |

Key observation: the **private** `queryCheckpoints(..., checkpointName, checkpointProperties, ...)`
in both readers **already accepts and combines** name AND properties into the request query
params. Only the public wrappers artificially restrict callers to "name OR properties,
never both".

## Problems this causes

### Problem 1 — `listDomainMajorVersionMeasurements`: no multi-value property filter, no combined pagination

`executionIDs` is a *set*, but `getCheckpointsByPropertiesPage` accepts a single
property map and filters with equality only. There is no way to say
`executionID IN (id1, id2, ...)` in one request. So we must:

- issue one request **per executionID**, and
- because per-request pagination is meaningless once results from several executionIDs are
merged, exhaustively page each executionID (loop until `hasMore == false`), then combine.

This means the incoming `limit`/`offset` cannot be honoured for the executionID path
(we currently ignore them and return the full combined set with `hasMore = false`), and we
pay N sequential round-trips for N executionIDs.

### Problem 2 — `listDomainMajorVersionFinalMeasurements`: cannot combine name + property

We need "the checkpoint(s) named `domainFinalCheckpointName` **and** having
`executionID = X`". No public method combines name and properties, even though the private
`queryCheckpoints` already builds both params. As a result the method currently **ignores**
`executionIDs` entirely and filters by name only. This is a functional gap, not just an
efficiency one.

## Requested reader changes

Ordered by priority.

1. **Public method combining name + properties.** Expose what `queryCheckpoints` already
does internally, e.g.:

```scala
def getCheckpointsPage(
checkpointName: Option[String] = None,
checkpointProperties: Map[String, String] = Map.empty,
pageSize: Int = 10,
offset: Long = 0,
includeProperties: Boolean = false
): F[RequestResult[PaginatedResponse[...]]]
```

This alone unblocks Problem 2 and lets the final-measurements query filter by name and
`executionID` together.

2. **Multi-value / `IN`-list property filtering.** Allow a property to match any of several
values in a single request, e.g. `Map[String, Set[String]]` or a dedicated
`executionIDs` filter. This removes the N-requests-per-set workaround in Problem 1 and
restores meaningful server-side pagination across the whole matching set.

3. **Server-side combined pagination.** With (1) and (2), a single request returns a
correctly paginated page over the full filtered set, so `limit`/`offset` can be honoured
for the executionID path instead of being ignored.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.