Pagination doesn't work when querying a GSI with prefixed sort key
- Dominant language
- Kotlin
- Stars
- 90
- Forks
- 42
- Avg merge
- 46m
- Merged PRs (30d)
- 1
Description
We encountered a failing non-null assertion in tempest's `Prefixer` when performing a query on a GSI using a prefixed sort key, once the result set grew large enough to require pagination, causing our service to be unable to read from the database.
Analysis by @jialiang-cash found that the `LastEvaluatedKey` object returned from Dynamo only includes the `partition_key` and `secondary_index_keys` fields. This caused an issue in `decodeOffset` when Tempest tries to remove the prefixes on both `partition_key` and `sort_key`, resulting in a null value exception because no `sort_key` exists.
Stack trace:
```
java.lang.IllegalArgumentException: Required value was null.
at app.cash.tempest.internal.ReflectionCodec$Prefixer.removePrefix(Codec.kt:137)
at app.cash.tempest.internal.ReflectionCodec.toApp(Codec.kt:92)
at app.cash.tempest2.internal.DynamoDbQueryable.decodeOffset(DynamoDbQueryable.kt:153)
at app.cash.tempest2.internal.DynamoDbQueryable.toQueryResponse(DynamoDbQueryable.kt:114)
at app.cash.tempest2.internal.DynamoDbQueryable.access$toQueryResponse(DynamoDbQueryable.kt:39)
at app.cash.tempest2.internal.DynamoDbQueryable$Sync.query(DynamoDbQueryable.kt:68)
at app.cash.tempest2.internal.LogicalDbFactory$SecondaryIndexFactory$secondaryIndex$1.query(LogicalDbFactory.kt)
at app.cash.tempest2.Queryable$DefaultImpls.query$default(Query.kt:27)
```
This is the assertion that's failing:
https://github.com/cashapp/tempest/blob/3f520efd7a9f1c5b341ea51a7efcb403e609cee0/tempest-internal/src/main/kotlin/app/cash/tempest/internal/Codec.kt#L137
Contributor guide
Research direction
Start in tempest-internal/src/main/kotlin/app/cash/tempest/internal/Codec.kt at Prefixer.removePrefix, then trace decodeOffset and toQueryResponse in tempest2/internal/DynamoDbQueryable.kt. Reproduce a paginated GSI query with a prefixed sort key and inspect the LastEvaluatedKey shape. Done means pagination no longer fails when the returned key lacks sort_key.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kotlin
- Domain
- database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100