awslabs / awslabs/aws-sdk-rust

High level DynamoDB client (or ODM)

Open
#70 5 comments 58 reactions 0 assignees View on GitHub
high-level-library p2
Dominant language
Rust
Stars
3.3k
Forks
290
Avg merge
1d 12h
Merged PRs (30d)
3

Description

[aws-ddb-best-practices]: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html
[single-table-design]: https://www.alexdebrie.com/posts/dynamodb-single-table/

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue, please leave a comment

### Terminology

[ODM](https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping#Object-oriented_databases) - object-document mapper

### Problem
There currently is a low-level DynamoDB client implemented in the SDK, it works with opaque `AttributeValue` types and direct DynamoDB APIs
https://github.com/awslabs/aws-sdk-rust/blob/7e43b19fd6fcc753bf5ceff4b2f5d13f6db799d8/sdk/dynamodb/src/model.rs#L6206-L6277

This low-level SDK crate provides no convenience APIs (batteries) that simplify the regular idiomatic ([Best Practices?](aws-ddb-best-practices), [single table design?](single-table-design)) usage of DynamoDB.

Working with raw `AttributeValue`s, and ad-hoc implementing common workflows is very inconvenient and error-prone.

## Solution

I propose we add a new crate that wraps `aws-sdk-dynamodb` low-level library and exposes the following "batteries-included" APIs (the list can be extended):

### ODM

Implement serialization and deserialization (i.e. object-document mapping) of strongly-typed structs **and** enums (both plain and [discriminated unions](https://en.wikipedia.org/wiki/Tagged_union)) into `aws_sdk_dynamodb::AttributeValue` via proc macros.

We can use `serde` to do the bulk of the job. I recommend taking over the job done in [`serde_dynamo`](https://docs.rs/serde_dynamo/2.3.0/serde_dynamo/) crate, and also learn the approaches [`dynomite`](https://docs.rs/dynomite/0.10.0/dynomite/) crate does.

The latter crate is more popular, but it is not very actively maintained. However, from my viewpoint, abusing `serde` as much as possible would be a better approach than implementing proc macros for converting between `AttributeValue` and strongly-typed structs and enums by hand, but that's debatable.

### Condition and update expression builder

[Condition expressions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html) are used in queries and scans, and [update expressions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.UpdateExpressions.html) are used in update operations, and they both use custom DynamoDB syntax.
It's okay to use raw strings with the standard Rust `format!()` macro for simple cases, but sometimes expressions are very dynamic and the expression might depend on lots of different variables and conditions.

Building raw condition expression syntax dynamically is very error-prone, the high-level wrapper crate should expose builders for expressions.
See TypeScript's implementation of this concept in [`@aws/dynamodb-expressions`](https://www.npmjs.com/package/@aws/dynamodb-expressions) package on `npm`.

### Projection expression utilities

Add some methods, maybe proc macros to generate the types that represent a [projection of different combinations of attributes](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ProjectionExpressions.html).
This requires some more thorough design, but the core problems to solve here:
- Prevent the usage of raw strings and raw syntax for building the projection expression
- Prevent working with raw `AttributeValue`

The API might look something like:

```rust
#[derive(Serialize, Deserialize, Projections)]
struct UserRecord {
partition_key: String,
sort_key: String,

#[project(Projection1)]
name: String,

// Come up with some syntax for nested properties projection (e.g. `ProjectionName = "")
#[project(Projection1, Projection2, Projection3 = "[0]")]
departments: Vec,

#[project(Projection1, Projection2)]
birth_date: chrono::NaiveDateTime,
}

// such that `#[derive(Projections)]` generates the following code:

#[derive(Serialize, Deserialize, Projection)]
struct Projection1 {
name: String,
departments: Vec,
birth_date: chrono::NaiveDateTime,
}

#[derive(Serialize, Deserialize, Projection)]
struct Projection2 {
departments: Vec,
birth_date: chrono::NaiveDateTime,
}

#[derive(Serialize, Deserialize, Projection)]
struct Projection3 {
// 0-th element of the original projected departments array
departments: String,
}

// where #[derive(Projection)] implements the `Projection` trait

impl Projection for Projection1 {
const PROJECTION_EXPRESSION: &'static str = "name, department, birth_date"
}

impl Projection for Projection2 {
const PROJECTION_EXPRESSION: &'static str = "departments, birth_date"
}

impl Projection for Projection3 {
const PROJECTION_EXPRESSION: &'static str = "departments[0]"
}
```

### Pagination utilities

Implement methods for streaming pagination (see [`dynomite::DynamoDbExt`](https://docs.rs/dynomite/0.10.0/dynomite/trait.DynamoDbExt.html) to learn about existing implementations).

### Other utilities for best practices and common workflows

Implement helper utilities according to [AWS docs for DynamoDB best practices](aws-ddb-best-practices) and [single table design](single-table-design).

For example, we've implemented some proc macros for segmented identifiers (described in single table design) in our private repo. We plan to open-source this code, and we may do it earlier to facilitate the development of the high-level DynamoDB crate.

So the list might also include:

- Utilities for working with segmented attributes (see [Alex DeBrie talking about this concept at re:Invent](https://youtu.be/DIQVJqiSUkE)).

### Additional context

This issue is nowhere an exhaustive description of the desired design for the high-level wrapper crate for `aws_sdk_dynamodb`, the ideas should be refined and probably extended. However, I think it might be a good starting point to begin the discussion and initiate the work on the MVP subset for the planned high-level APIs (e.g. start with only ODM feature and iterate from that next).
We may decide to separate the described crate to other repo and split the planned features described here into more fine-grained issues if this makes sense to the maintainers.

Waiting for your feedback!

Contributor guide

Open the contributing guide

Research direction

Start with the low-level DynamoDB model in sdk/dynamodb/src/model.rs, especially the referenced AttributeValue definitions, then compare the approaches in serde_dynamo and dynomite. Before implementation, refine the broad proposal into an agreed MVP, such as the ODM subset; done should mean that scoped API is specified and covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
api, backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.