awslabs / awslabs/data-solutions-framework-on-aws
Add option to DataLakeCatalog/DataCatalogDatabase that crawlers use manually defined table
- Dominant language
- TypeScript
- Stars
- 147
- Forks
- 31
- PR merge metrics
- No merged PRs in 30d
Description
DataLakeCatalog/DataCatalogDatabase should have the option of manually setting the tables for the crawler as parameters. There are several use cases that require a manually created catalog table.
- You want to choose the catalog table name manually and not rely on the catalog table naming algorithm
- Reuse the table later in the stack (e.g., in a lambda to query the table)
**See:** https://docs.aws.amazon.com/glue/latest/dg/tables-described.html#update-manual-tables
Contributor guide
Research direction
Start with the DataLakeCatalog and DataCatalogDatabase entry points and read the linked AWS Glue documentation on manually defined tables. Determine how a caller should provide table definitions and how those tables can be reused later in the stack, such as by a Lambda. Done means the crawler accepts manually specified catalog tables and the resulting tables can be referenced by name.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100