cashapp / cashapp/tempest

Pagination doesn't work when querying a GSI with prefixed sort key

Open
#121 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Kotlin
Stars
90
Forks
42
Avg merge
46m
Merged PRs (30d)
1

Description

We encountered a failing non-null assertion in tempest's `Prefixer` when performing a query on a GSI using a prefixed sort key, once the result set grew large enough to require pagination, causing our service to be unable to read from the database.

Analysis by @jialiang-cash found that the `LastEvaluatedKey` object returned from Dynamo only includes the `partition_key` and `secondary_index_keys` fields. This caused an issue in `decodeOffset` when Tempest tries to remove the prefixes on both `partition_key` and `sort_key`, resulting in a null value exception because no `sort_key` exists.

Stack trace:
```
java.lang.IllegalArgumentException: Required value was null.
at app.cash.tempest.internal.ReflectionCodec$Prefixer.removePrefix(Codec.kt:137)
at app.cash.tempest.internal.ReflectionCodec.toApp(Codec.kt:92)
at app.cash.tempest2.internal.DynamoDbQueryable.decodeOffset(DynamoDbQueryable.kt:153)
at app.cash.tempest2.internal.DynamoDbQueryable.toQueryResponse(DynamoDbQueryable.kt:114)
at app.cash.tempest2.internal.DynamoDbQueryable.access$toQueryResponse(DynamoDbQueryable.kt:39)
at app.cash.tempest2.internal.DynamoDbQueryable$Sync.query(DynamoDbQueryable.kt:68)
at app.cash.tempest2.internal.LogicalDbFactory$SecondaryIndexFactory$secondaryIndex$1.query(LogicalDbFactory.kt)
at app.cash.tempest2.Queryable$DefaultImpls.query$default(Query.kt:27)
```

This is the assertion that's failing:
https://github.com/cashapp/tempest/blob/3f520efd7a9f1c5b341ea51a7efcb403e609cee0/tempest-internal/src/main/kotlin/app/cash/tempest/internal/Codec.kt#L137

Contributor guide

Open the contributing guide

Research direction

Start in tempest-internal/src/main/kotlin/app/cash/tempest/internal/Codec.kt at Prefixer.removePrefix, then trace decodeOffset and toQueryResponse in tempest2/internal/DynamoDbQueryable.kt. Reproduce a paginated GSI query with a prefixed sort key and inspect the LastEvaluatedKey shape. Done means pagination no longer fails when the returned key lacks sort_key.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kotlin
Domain
database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.