awslabs / awslabs/data-solutions-framework-on-aws

Add option to DataLakeCatalog/DataCatalogDatabase that crawlers use manually defined table

Open
#490 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
147
Forks
31
PR merge metrics
No merged PRs in 30d

Description

DataLakeCatalog/DataCatalogDatabase should have the option of manually setting the tables for the crawler as parameters. There are several use cases that require a manually created catalog table.

- You want to choose the catalog table name manually and not rely on the catalog table naming algorithm
- Reuse the table later in the stack (e.g., in a lambda to query the table)

**See:** https://docs.aws.amazon.com/glue/latest/dg/tables-described.html#update-manual-tables

Contributor guide

Open the contributing guide

Research direction

Start with the DataLakeCatalog and DataCatalogDatabase entry points and read the linked AWS Glue documentation on manually defined tables. Determine how a caller should provide table definitions and how those tables can be reused later in the stack, such as by a Lambda. Done means the crawler accepts manually specified catalog tables and the resulting tables can be referenced by name.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.