aws-amplify / aws-amplify/amplify-cli

Using Athena with Firehose, Glue etc.

Open
#2,949 21 comments 5 reactions 1 assignee Claimed by @swaminator View on GitHub
analytics feature-request pending-review
Dominant language
TypeScript
Stars
2.9k
Forks
825
Avg merge
11d 23h
Merged PRs (30d)
2

Description

I've been trying to get an analytics/stats dashboard developed for our platform which is built on Amplify.

I've tried several attack angles and read countless articles which all try to get to the same end result; Being able to query data in Athena using ANSI SQL to create stats/charts.

Our data is stored in DynamoDB and I'm hoping for some pointers on best practice, and maybe some boilerplate code to reliably get data from DynamoDB to S3 on create/update. I've played with some Python code, but I cannot really figure out how to get that into S3 in Parquet format, and saving it to JSON in S3 and then have a Glue job pick that up and convert to Parquet seems unnecessarily complex.

The goal is a near-real-time mirror of some of our DynamoDB data in Parquet format in S3 that can then be queried through the Athena API. I read one article that described how to create custom GraphQL queries to query Athena and that seems very promising, but my main struggle right now really is to get data out of DynamoDB and into S3 in Parquet format for optimized storage and query performance.

I saw that Kinesis Firehose features were added to Amplify recently, but I am unsure if those additions will be helpful to this challenge in any way?

If anyone has succeeded in this and would share their adventure of if AWS devs have information in relation to this challenge, I'd appreciate it very much.

**What AWS Services are you utilizing?**

Amplify.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.