[Enhancement] Able to alter existing table when statement reaches query API limits
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 324
- PR merge metrics
- No merged PRs in 30d
Description
## Background
It is pretty common that there will have new keys added to an event schema over the time. User will have to add these new keys as optional keys to the schema, alter the Athena table to reflect new keys and allow historical data to use new schema as well.
StreamAlert provides command line to alter the table based on the new schema by running following command.
```
manage.py athena rebuild-partitions --bucket s3.bucket.name --table-name my_athena_table
```
However, [the maximum allowed query string length in Athena API call is 262144 bytes](https://docs.aws.amazon.com/athena/latest/ug/service-limits.html). In [current implementation](https://github.com/airbnb/streamalert/blob/master/stream_alert_cli/athena/handler.py#L137), StreamAlert will save the whole query into a txt file if the length exceeds the limit. And then the user has to run the query from Athena console manually. It is not ideal.
## Desired Change
Slice query string based on the length limit and loop through the query strings to alter table.
Contributor guide
Research direction
Start in stream_alert_cli/athena/handler.py at the current query-length handling used by manage.py athena rebuild-partitions. Check the Athena API limit and existing command flow, then verify that oversized changes are processed in bounded query strings and that the table is altered for the full schema.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cli, cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100